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Abstract 

This  study  develops  a  high  level,  unifying  taxonomy  for  Modeling,  Simulation,  and 
Analysis  (MS&A)  products  for  the  Air  Force  Materiel  Command  (AFMC).  AFMC  is 
concerned  that  limited  resources  are  being  expended  on  duplicative  MS&A  efforts.  No 
mechanism  exists  that  would  confirm  or  deny  this  concern,  so  it  was  suggested  that  a 
database  could  be  developed  to  catalog  and  track  AFMC’s  MS&A  inventory.  First,  it  was 
necessary  to  determine  the  information  that  a  decision  maker  needs  to  select  a  suitable 
MS&A  product.  Potential  traits  and  characteristics  were  identified  through  review  of 
current  regulatory  guidance,  interviews  with  MS&A  users,  and  a  study  of  the  current 
literature.  Using  the  collected  information,  a  survey  was  developed  and  distributed  to  40 
members  of  the  Modeling  and  Simulation  Technical  Planning  Integrated  Product  Team 
(M&S  TPIPT).  Survey  results  provided  the  foundation  for  developing  a  limited  prototype 
database.  This  prototype  was  tested  to  ascertain  the  retrieval  performance  of  the 
cataloging  system.  The  test  results  failed  to  confirm  the  retrieval  capability,  but  the  test 
participants  believed  that  cataloging  AFMC’s  MS&A  inventory  would  have  great  benefit. 
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AIR  FORCE  MATERIEL  COMMAND  (AFMC) 
MODELING,  SIMULATION,  AND  ANALYSIS  (MS&A) 
INTERACTIVE  DATABASE 


I.  Background  and  Problem  Statement 


Chapter  Overview 

The  use  of  Modeling,  Simulation,  and  Analysis  (MS&A)  software,  in  support  of 
Air  Force  activities  at  all  levels,  is  becoming  more  prevalent  every  year.  In  response  to 
this  ever  increasing  use,  a  growing  concern  is  that  we  ensure  our  MS&A  resources  are 
expended  wisely.  At  the  present  time,  there  is  no  centralized  inventory  of  MS&A 
software,  which  leads  to  difficulties  in  trying  to  control  our  MS&A  expenditures. 

This  chapter  covers  the  general  issues  surrounding  MS&A  software,  provides  a  concise 
statement  of  the  problem,  states  the  research  question  and  objectives  to  be  addressed, 
presents  an  encapsulated  version  of  the  research  design,  and  describes  the  expected 
resultants  of  the  research,  namely  a  classification  system  and  prototype  database. 

Problem  Statement 

The  general  issue  facing  the  Air  Force  is  the  expenditure  of  approximately  60  to  70 
million  dollars  each  year  on  the  development  of  MS&A  software,  according  to  Maj  Steve 
Chimelski,  HQ  AFMC/XRX  (3).  A  1  Mar  1993  DoD  Inspector  General  (IG)  audit  on 
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Research  Problem  Emphasis 

The  specific  problem  is  to  develop  a  classification  system  using  identifying 
characteristics  of  MS&A  products.  A  database  could  then  be  based  on  that  system  and 
used  to  facilitate  the  crossflow  of  information  between  MS&A  user  organizations.  A 
database  of  such  information  would  function  as  a  repository  of  existing  MS&A  products 
that  could  be  relevant  to  a  particular  problem  or  question.  For  example,  a  program 
manager  for  a  new  developmental  fighter  aircraft  might  be  interested  in  any  models  that 
deal  with  aerodynamic  simulations  of  supersonic  aircraft.  A  user-friendly,  interactive 
database  would  be  the  vehicle  to  provide  the  desired  crossflow  of  information.  At  a 
minimum,  the  database  should  provide  the  title  of  the  software,  brief  description,  and  a 
point  of  contact  for  further  information.  The  database  would  also  allow  program 
managers  in  all  parts  of  the  country  to  cross-check  any  current  MS&A  software 
apphcations  for  use  in  their  programs.  This  would  save  both  time  and  funds  by  eliminating 
duplicate  development  efforts.  HQ  AFMC/XRX  took  the  lead  in  attempting  to  develop 
such  a  database.  They  started  with  a  massive  data  collection  effort  to  capture  the  various 
types  of  MS&A  software  found  across  the  Air  Force.  Unfortunately,  the  development 
effort  has  languished  due  to  manpower  and  time  constraints. 

We  must  determine  the  characteristics  and  questions  that  must  be  addressed  in 
order  for  a  decision  maker  to  decide  whether  to  create  a  MS&A  program  or  use  an 
existing  one.  Some  of  the  characteristics  that  may  be  critical  to  decision  makers  include 
the  modehng  technique  employed  in  the  program,  the  duration  of  the  simulation  run  (ran 
time),  the  user-friendliness  of  the  program,  or  the  learning  that  must  take  place  in  order  to 
use  the  program. 

The  investigative  questions  which  must  be  addressed  are: 

-  What  traits  differentiate  one  MS&A  program  from  another? 

-  Which  traits  are  most  useful  to  analysts  in  selecting  MS&A  software? 
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-  Can  a  database  based  on  a  cataloging  system  which  uses  such  traits  provide 
acceptable  retrieval  performance  for  users? 

Unless  we  are  able  to  define  a  link  between  MS&A  software  and  its  application, 
the  best  we  can  hope  for  is  a  massive  inventory  list  of  every  MS&A  program  in  the  Air 
Force.  This  list  would  have  little  practical  application  for  program  managers  and  would 


MUJ 


HIHj 


should  permit  the  Air  Force  to  realize  savings  in  the  annual  MS&A  budget  by  reducing  or 
eliminating  unnecessary  MS&A  software  development  or  purchases. 

Summary 

This  chapter  establishes  the  focus  of  this  research  effort.  With  an  increasing  use 
and  dependence  on  MS&A  software  coupled  with  an  ever  decreasing  budget,  the  Air 
Force  must  develop  some  method  of  matching  current  needs  to  capabilities.  An  effective 
database  system  of  MS&A  software  as  described  in  this  chapter  would  provide  the  means 
for  analysts  to  match  their  current  simulation  needs  with  existing  MS&A  capabilities.  This 
thesis  proposes  a  research  objective  and  methodology  which  will  address  these  needs. 
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Before  pursuing  the  research  objective  of  identifying  MS&A  traits  and  their 
differentiation  usefulness,  we  should  review  the  relevant  information  concerning  MS&A, 
databases,  and  associated  topics. 


informatioini  Systems 

The  general  premise  of  this  study  is  to  provide  program  managers,  analysts,  and 
various  other  MS&A  users  a  tool  which  can  be  used  to  provide  an  informational  crossflow 
and  to  reduce  the  expenditure  of  funds  due  to  redundant  development  efforts.  The  intent 
is  for  users  to  be  able  to  query  an  information  system  to  ascertain  whether  or  not  a  MS&A 
tool  already  exists  to  answer  their  potential  problem  or  question.  An  information  system 
is  defined  as  “a  set  of  procedures  that  collect  (or  retrieve),  process,  store  and  disseminate 
information  to  support  decision  making  and  control”  (10:  5).  There  are  various  types  of 
information  systems  to  support  different  levels  of  management  as  reflected  in  Figure  1. 

Given  the  operational  premise  of  this  information  system,  we  can  conclude  that  we 
are  basically  dealing  with  a  Transaction  Processing  System  (TPS)  because  we  are 
operating  at  an  elementary  level  of  querying  data  to  answer  a  specific  question.  One  way 
to  approach  implementing  this  TPS  is  by  means  of  a  database.  A  database  is  defined  as  a 
computerized  collection  of  stored  operational  data  that  serves  the  needs  of  multiple  users 
within  one  or  more  organizations”  (25: 4).  Our  needs  fit  neatly  within  the  parameters  of 
the  above  definition.  We  need  a  system  that  can  function  as  a  centralized  resource  of 
stored  information  that  can  answer  queries  concerning  MS&A  tools.  This  system  must  be 
able  to  provide  this  information  to  a  wide  range  of  users  across  AFMC  at  its  many  diverse 
locations.  In  order  to  answer  these  needs,  we  need  to  create  an  information  system.  This 


system  will  most  likely  be  in  the  form  of  a  database,  because  this  is  the  preferred  method 
of  delivery  from  the  AFMC  MS&A  community’s  perspective  (3). 
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Figure  1.  Hierarchy  of  Information  Systems  (10:  7,  adapted). 


Database  Development 

Database  development  follows  a  series  of  steps  intended  to  provide  a  useful 
product  that  meets  the  using  community’s  needs.  The  steps  shown  in  Figure  2  and 
discussed  below  are  a  compilation  of  material  from  Teorey  and  Fry’s  text  and  class  notes 
from  an  AFIT  class,  IMGT  699,  Introduction  to  Database  Systems  taught  by  Lt  Col  Chris 
Arnold.  Figure  2  shows  the  four  database  design  steps  in  a  graphical  format. 

Requirements  Formulation  and  Analysis:  We  start  requirements 
formulation  by  exploring  the  underlying  purpose  and  reason  for  the  database.  We  look  at 
the  body  of  data  in  question,  the  views  and  goals  of  the  users,  the  support  of  senior 
management,  and  the  organizational  structure.  This  information  is  often  collected  through 
observation  of  the  workplace  and  interviews  with  users  and  managers.  Additional  sources 
of  information  that  should  be  reviewed  are  management  reports,  documentation,  operating 
instructions,  policy  and  guidance,  and  regulations  that  may  impact  the  development  effort. 
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L  Basic  Database  Design  Steps  (25;  26,  Adapted). 


Coiniceptusl  Desigmi  “The  conceptual  design  concerns  itself  with  the  description 
and  synthesis  of  diverse  user’s  information  requirements  into  a  preliminary  database 
design”  (25:  27).  The  data  collected  in  the  first  step  of  the  design  process  provides  the 
foundation  for  designing  a  high  level  conceptual  representation  or  conceptual  schema  of 
the  database.  This  representation  is  built  by  using  an  Entity-Relationship  (ER)  model. 
“The  ER  model  describes  the  data  as  entities,  relationships,  and  attributes”  (6: 42).  We 
map  the  physical  world  (i.e.  the  collected  data)  into  a  conceptual  schema  by  using  the  ER 
model.  Let’s  say  we  wish  to  model  a  college’s  student  scheduling  process.  Information  is 
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kept  on  every  student  consisting  of  name,  social  security  account  number  (SSAN), 
resident  address,  permanent  address,  home  phone,  and  academic  program.  The  school  has 
many  departments  with  instructors  assigned  and  courses  for  which  the  department  is 
responsible.  How  would  we  depict  this  scheduling  process  in  an  ER-model?  First,  we 
need  to  identify  the  entities  involved.  An  entity  is  “a  ‘thing’  in  the  real  world  with  an 
independent  existence”  (6: 43).  Therefore,  an  entity  could  be  a  person,  employee,  school, 
house,  department,  course,  or  some  other  physical  object.  The  describing  characteristics 
of  an  object  are  called  attributes.  Thus,  the  attributes  such  as  a  person’s  name,  SSAN, 
home  address,  and  home  phone  could  be  used  to  describe  an  entity  called  student.  After 
the  entities  have  been  identified  and  described  (i.e.  given  attributes),  then  we  need  to  look 
at  how  the  entities  interact  among  themselves.  These  interactions  are  referred  to  as 
relationships.  Now  that  we  have  a  basic  understanding  of  the  ER-model;  we  can  proceed 
to  develop  the  student  scheduling  process.  Figure  3  shows  what  the  student  scheduling 
process  would  look  like  as  an  ER-model.  The  student  entity  has  a  relationship  with  the 
section  entity.  The  student  can  enroll  in  many  sections  (thus  an  N  on  the  section  side)  and 
each  section  can  have  many  students  (thus  an  M  on  the  student  side).  The  student  also 
has  a  relationship  with  the  department  in  the  sense  that  a  student  can  participate  in  only 
one  degree  program  offered  by  a  department  (thus  a  1  on  the  department  side),  but  a 
degree  program  may  have  many  smdents  enrolled  (thus  an  N  on  the  student  side).  This  is 
not  to  say  that  the  department  can  not  or  does  not  have  numerous  degree  programs,  it 
merely  means  that  each  student  can  only  be  enrolled  in  one  degree  program  at  a  time. 

Note  the  double  lines  between  the  student  and  department.  This  double  line  emphasizes 
that  the  relationship  between  the  two  entities  is  a  “total  participation”  (6:  54).  This  means 
that  every  student  is  associated  with  a  department  and  every  department  has  students. 
Contrast  this  with  the  relationship  between  sections  and  courses.  Each  and  every  section 
offering  relates  to  only  one  course.  However,  if  a  course  is  not  offered  that  semester,  the 
course  will  have  no  sections.  Thus,  each  course  could  have  one  or  more  sections  (so  we 
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have  the  N  on  the  sections  side)  yet  each  section  is  only  related  to  one  course  (so  we  have 
a  1  on  the  course  side).  A  course  does  not  have  to  be  offered  each  semester  and,  as  a 
consequence,  it  would  not  have  a  section  (thus  a  single  line  from  the  course  side).  This 
single  line  represents  a  “partial  participation”  (6;  54).  Partial  participation  means  that 
some  part  or  subset  of  the  courses  entity  is  related  to  the  sections  entity  for  a  particular 
term  or  semester.  This  example  presents  some  of  the  considerations  and  conventions 
involved  in  consmicting  an  ER-modeL  For  the  MS&A  database,  we  will  need  to  identify 
the  appropriate  entities  with  their  associated  attributes  and  the  relationships  between 
entities.  Once  this  is  completed,  we  are  ready  to  design  the  database. 
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data  which  represent  a  collection  of  related  information  or,  from  an  ER-model  viewpoint, 
each  individual  row  would  be  an  entity  with  its  associated  attributes.  A  physical  example 
of  a  table  would  be  the  Dayton  area  phone  book.  Each  individual  listing  of  person  or 
business,  associated  address,  and  corresponding  telephone  number  would  represent  the 
information  contained  in  an  individual  row  and  all  of  the  rows,  in  total,  would  establish  the 
table  (i.e.  the  Dayton  area  phone  book). 

In  a  relational  database,  there  are  no  explicit  links  between  tables.  Instead,  a  query 
language  is  used  to  make  connections  between  tables.  Consider  the  phone  book  example. 
Let  that  be  one  table  and  a  second  table  consists  of  customers  of  a  business.  A  row  in  the 
second  table  contains  the  name,  phone  number,  and  account  balance  for  each  customer. 
Now  the  address  for  each  customer  can  be  retrieved  by  matching  customer  phone  numbers 
with  the  corresponding  phone  number  in  the  phone  book. 

Once  the  relational  model  is  developed,  we  can  proceed  to  defining  data 
dictionary.  A  data  dictionary  is  defined  as  “a  list  of  all  database  tables  and  fields”  (26: 

329).  The  database  management  system  (DBMS)  will  automatically  create  this  list  which 
stores  the  name,  lengths,  type  of  data  (i.e.  text,  number,  date,  etc.),  and  other  information 
which  provides  an  unifying,  consistent  format  through  out  the  database  application. 
However,  another  function  of  the  data  dictionary  is  to  resolve  any  ambiguities  in  what  the 
data  represents.  For  instance,  if  field  is  entitled  WAGE,  what  value  should  be  entered? 
Obviously  one  would  enter  a  number  value,  but  determining  that  number  is  where  possible 
inconsistency  could  exist.  The  worker  may  very  well  view  WAGE  as  the  amount  one 
brings  home  in  a  paycheck  after  taxes.  Conversely,  the  IRS  views  WAGE  as  the  amount 
paid  to  the  employee  before  taxes.  Finally  the  employer  may  view  WAGE  as  the  sum  of 
pay,  benefits,  allowances,  and  contributions  made  on  the  behalf  of  the  employee.  Thus  the 
field  needs  to  be  defined  to  help  prevent  any  ambiguity  and  inconsistency.  Once  the 
relationships  and  data  dictionary  are  defined,  we  can  start  building  the  physical  database 
and  inputting  the  data.  From  here  we  would  progress  to  the  last  step  of  Figure  2. 
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PhySDeaS  Desioin;  The  physical  design  covers  three  main  categories;  stored 
record  format  design,  clustering  analysis  and  design,  and  access  path  design  (25: 28). 

Each  of  these  looks  at  detailed,  technical  aspects  of  the  database  design  such  as 
partitioning  of  data  items  to  different  physical  locations  (i.e.  a  distributed  database), 
memory  allocation,  central  processor  unit  (CPU)  speed  and  block  size  calculations  for 
data  retrieval,  and  other  issues  which  are  outside  the  design  of  this  prototype  database  and 
this  thesis.  The  prototype  database  is  being  used  as  a  proof  of  concept  of  the  MS&A 
traits  and  their  usefulness;  not  as  a  completed,  operational  database. 

T©StDing°  The  last  step  in  the  process  would  be  a  test  of  the  prototype  to  gain  an 
indication  of  how  well  the  prototype  performs.  Although  this  is  not  a  formalized  step  of 
the  process  according  to  Figure  2,  it  is  nonetheless  an  important  one.  The  review  of  the 
prototype  by  current  simulation  users  will  define  the  applicability  of  the  MS&A  traits. 


Literator®  Sources  @f  MS&A  Traits 

From  an  extensive  review  of  the  current  literature,  I  was  unable  to  find  much 
concerning  any  taxonomies  describing  MS&A  attributes  in  the  commercial  sector.  T.  I. 
Oren  wrote  an  article  addressing  the  attributes  of  cognizant  simulation  which  is  a 
specialized  form  of  simulation  that  uses  artificial  intelligence  (AI)  developed  from  expert, 
rule-based,  or  knowledge-based  applications.  The  taxonomy  is  split  into  two  distinct 
segments:  cognizant  simulation  and  cognizant  environments.  The  cognizant  simulation  is 
further  delineated  into  six  major  categories  which  are  “numerical  simulation  with  nested 
neural  nets,  knowledge-based  simulation,  qualitative  simulation,  multiparadigm  simulation, 
simulation  with  a  nested  knowledge-based  system,  and  knowledge-based  system  with  a 
nested  simulation  system”  (15: 296).  The  cognizant  environment  is  delineated  into  four 
categories  which  are  “cognizant  (intelligent)  interfaces;  cognizant  environments  for  single 
paradigm  and  multiparadigm  simulations  (for  models,  model  parameters,  experiments. 
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programs,  quality  assurance,  and  for  AI  components);  cognizant  environments  with  a 
nested  knowledge-based  system;  and  comprehensive  cognizant  environments”  (15:  302). 
Oren  believes  that  his  taxonomy  is  useful  for  several  reasons  because  “one  can  perceive 
the  unity  of  the  field,  one  can  classify  the  current  achievements,  and  one  can  systematically 
explore  promising  new  areas”  (15:  293).  The  categories  used  in  Oren’s  taxonomy  lead  to 
the  potential  idea  of  characterizing  all  simulation  products  based  upon  their  code  or 
internal  structure. 

Jenny  Preece  and  H.  Dieter  Rombach  have  developed  a  taxonomy  that  provides 
“a  framework  to  describe  the  approaches  and  techniques  used  in  current  software 
engineering  (SE)  and  human-computer  interaction  (HCI)  measurement”  (18: 555).  Often 
both  of  these  areas  strive  to  address  similar  areas  of  concern,  however,  they  often  do  so 
with  conflicting  methodologies.  For  instance,  a  system  may  have  had  a  streamlined  and 
compact  method  for  entering  data  (an  SE  approach),  but,  if  the  actual  user  finds  the  input 
method  cumbersome  or  overly  difficult,  then  the  software  has  failed  from  an  HCI 
perspective.  The  taxonomy  they  have  developed  attempts  to  unify  the  two  disciplines  by 
identifying  common  characteristics  of  any  study.  The  taxonomy  looks  at  four  major 
dimensions  of  any  SE  or  HCI  study: 

(i)  the  goal  of  the  study  (what  is  being  looked  at  and  why?) 

(ii)  the  plan  of  the  study  (what  is  the  underlying  philosophy,  how  much  and  what 
kind  of  external  influence  is  brought  to  the  study  and  what  is  the  location  and 
design  of  the  study?) 

(iii)  the  study  methods  employed  (who  does  the  study,  what  do  they  do  and  when 
do  they  do  it?),  and 

(iv)  the  kind  of  techniques  that  are  used  (how  is  data  being  collected,  analyzed  and 
validated,  and  how  is  the  information  derived  from  it  being  both 
communicated  back  to  the  project  itself  and  reused  to  inform  future 
projects?).  (18:556) 


These  major  dimensions  can  be  further  refined  as  necessary  to  provide  the  required 
degree  of  detail  about  each  study.  Preece  and  Rombach  believe  that  their  taxonomy  can 
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provide  a  unifying  structure  for  these  two  diverse,  yet  intimately  related,  computer 
disciplines.  The  taxonomy  establishes  a  common  ground  to  provide  the  capability  of 
reusing  previously  completed  studies,  planning  for  future  research,  guiding  current  studies, 
and  facilitating  communications  between  the  two  disciplines. 

The  last  taxonomy  to  be  covered  is  one  developed  by  B.  W.  HoUocks.  Bollocks 
reviewed  the  application  of  simulation  in  manufacturing  within  the  United  Kingdom  (UK). 
The  focus  of  this  study  was  not  on  what  simulation  products  UK  manufacturing  employs 
but  in  what  areas  of  application  the  simulation  was  used  to  support.  Bollocks  identified 
14  applications  areas  and  queried  simulation  users  as  to  the  areas  in  which  they  currendy 
use  simulation  tools.  From  the  survey,  it  was  found  that: 

users  employ  simulation  in  on  average  6  (5.98)  of  the  areas.  (The  study  found  a 
minitTintTi  of  two.)  TWs  is  not  surprising  given  the  alliances  between  items  in  the 
list;  for  example,  capital  equipment  decisions  also  commonly  involve  plant  layout, 
line  balancing  is  associated  with  manning  levels,  and  material  control  rules  affect 
inventory  levels.  (9: 107) 

Based  upon  the  results,  Bollocks  grouped  the  14  application  areas  in  five  broad  areas: 
facilities,  productivity,  resourcing,  training,  and  operations.  Bollocks’  study  points  out 
that  even  if  a  taxonomy  can  be  established,  it  may  not  have  a  high  degree  of  resolution 
because  the  simulation  can  be  so  intertwined  in  their  application  areas. 

Even  given  the  few  examples  above,  I  was  unable  to  find  any  taxonomy  which 
attempted  to  draw  together,  into  one  unifying  framework,  all  of  the  different  aspects  of 
M&S.  This,  in  itself,  is  not  surprising.  Many  weU  known  simulation  packages  are  generic 
in  nature;  designed  to  address  many  potential  problems  with  a  formalized  approach. 
Examples  of  some  of  the  best  known  simulation  software  languages  are  SLAM,  GASP, 
SIMAN,  and  SMSCRIPT.  Conversely,  Air  Force  simulation  packages  are  often  designed 
to  address  a  specific  problem  or  application.  Many  of  the  different  sources  used  to 
identify  varying  characteristics  of  MS&A  are  discussed  below. 
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DoD  5000.59-Paa,  Modeling  and  Simulation  (M&S)  Master  Plan  (Draft). 

The  primary  source  describing  the  current  state  of  military  MS&A  guidance  is 
DoD  5000.59-Paa,  Modeling  and  Simulation  (M&S)  Master  Plan  (Draft),  January  1995 
prepared  by  the  Under  Secretary  of  Defense  (Acquisition  and  Technology).  The  plan  is 
the  definitive  source  for  policy  and  guidance  within  the  DoD  for  M&S  and  it  implements 
policy  in  DoD  Directive  5000.59;  establishes  DoD-wide  M&S  objectives;  provides  a 
comprehensive  framework  for  the  planning,  programming,  and  budgeting  of  M&S 
projects,  programs,  and  activities;  and  assigns  responsibilities  for  its  implementations”  (27: 
i).  To  oversee  these  activities  the  DoD  Executive  Council  for  Modeling  and  Simulation 
(EXCIMS)  was  established.  The  EXCIMS  formulated  a  direction  and  vision  for  M&S: 


Defense  modeling  and  simulation  will  provide  readily  available,  operationally 
valid  environments  for  use  by  DoD  Components: 

-  to  train  jointly,  develop  doctrine  and  tactics,  formulate  operational  plans,  and 
assess  warfighting  situations 

-  to  support  technology  assessment,  system  upgrade,  prototype  and  fuU-scale 
development,  and  force  structuring 

Furthermore,  common  use  of  these  environments  will  promote  a  closer 
interaction  between  the  operations  and  acquisition  communities  in  carrying  out 
their  respective  responsibilities.  To  allow  maximum  utility  and  flexibility,  these 
modeling  and  simulation  environments  will  be  constructed  from  affordable, 
reusable  components  interoperating  through  an  open  systems  architecture.  (27: 
2-1) 


A  visual  representation  of  the  EXCIMS  vision  is  reflected  in  Figure  4.  The 
EXCIMS  vision  offers  an  unifying  concept  which  ties  together  the  varying  levels  of  M&S, 
the  different  areas  of  application,  the  broad  range  of  user  organizations,  and  the  many 
perspectives  and  requirements  of  M&S.  This  leads  to  the  idea  that  M&S  can  be 
characterized  by  these  unifying  concepts.  For  instance,  the  scope  of  M&S  can  be 
represented  from  the  lowest  level  of  subsystem/component  tools  to  the  high  level 
theater/campaign  models.  More  will  be  discussed  about  the  hierarchy  (scope)  of  M&S 
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models  in  the  Systems  Acquisition  Manager’s  Guide  section  below.  The  functional  area  of 
application  suggests  that  M&S  could  be  defined  by  the  functional  orientation  of  the  model. 
Models  could  be  categorized  as  being  most  applicable  in  the  following  areas:  Education, 


on  those  plans,  and  rehearse  missions”  (27: 2-4).  Simulation  allows  training  to  be 
conducted  on  a  joint  level  without  physically  moving  forces  by  means  of  an  interactive, 
synthetic  battlefield.  Computer  generated  forces  can  interact  with  human  participants 
under  actual  field  conditions  using  operational  weapon  and  command  and  control  systems. 
Feedback,  both  real-time  and  after  action  reports,  can  be  used  to  refine  operational  plans, 
doctrines,  and  tactics,  to  assess  operational  deficiencies  or  to  define  unit  effectiveness  and 
readiness. 


Figure  5.  Four  Pillars  of  Military  Capability. 

Modernization,  the  second  pillar,  can  benefit  from  M&S  by  reducing  “the  lime, 
resources,  and  risks  of  the  acquisition  process”  and  to  “increase  the  quality  of  the  systems 
being  acquired”  (27:  2-5).  Proposed  weapon  systems  can  be  tested  for  their  impact  upon 
operational  and  logistics  systems.  M&S  can  provide  preliminary  test  results  of  a  weapon 


system  under  varying  environments  and  conditions  to  help  ensure  that  a  realistic  test  and 
evaluation  scenario  is  being  followed. 

The  next  pillar.  Force  Structure,  will  benefit  from  M&S  by  giving  “DoD  leadership 
a  powerful  arsenal  of  tools  to  analyze  alternative  DoD  force  structures  (27. 2-7).  Senior 


System/Subsystem  Component  (17:  3-16  and  3-17). 

The  Strategic/National  Military  Strategy  simulation  packages  model  the  highest 
level  of  political,  economic,  and  military  policies  and  concerns.  Policy  decisions  such  as 
force  structure  for  a  two-front  war,  impact  of  the  drawdown,  forward  basing 


agreements  or  alliances  would  be  addressed  at  this  level.  The  results  of  this  level  of 
simulation  are  strategic  in  nature  and  represent  a  duration  of  weeks,  months,  or  years. 


The  Theater/Campaign  level  models  a  wartime  scenario  addressing  various  aspects 
of  aerospace  power  within  theater(s)  of  operations.  The  defense  of  Europe  or  the 
repelling  of  a  North  Korean  attack  would  be  modeled  at  this  level.  Casualty  rates,  spares 
usage,  resupply  needs,  mission  effectiveness,  and  other  like  concerns  would  be  addressed. 
The  results  of  this  level  of  simulation  are  strategic  and  operational  in  nature  and  represent 
a  duration  of  days  to  weeks. 

The  Mission  level  models  various  aspects  of  aerospace  power  as  it  relates  to  a 
particular  mission.  A  bombing  run  on  a  heavily  defended  airfield  could  be  analyzed  to 
determine  casualties,  best  avenue  of  approach,  best  mix  of  aircraft  and  capabilities,  and 
probability  of  success  could  be  evaluated.  The  results  of  this  level  of  simulation  are 
operational  and  tactical  in  nature  and  represent  a  duration  of  hours. 

The  Engagement/Submission  level  models  a  specific  or  finite  set  of  assumptions  or 
criteria  to  determine  performance.  How  a  flight  of  fighter/bombers  fares  against  a  battery 
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hours  or  more.  The  comparison  in  Figure  7  shows  that  the  Army  uses  a  hierarchy  similar 
to  the  Air  Force  with  the  exception  of  not  having  a  Strategic/National  Military  Strategy 


level  (17:  4-6). 

As  shown  in  Figure  7,  the  lowest  level  of  the  Army  hierarchy  is  the  Engineering 


Figure  7.  Comparison  of  AF  versus  Army  Hierarchy. 


The  guide  views  the  user  community  as  being  divided  into  the  following  functional 
areas:  research  and  development:  test  and  evaluation;  analysis  and  production  and 
logistics.  Education,  training  and  operations  concentrates  on  the  re-creation  of  historical 
battles,  doctrine  and  tactics  development,  command  and  unit  training,  operational  planning 
and  rehearsal,  and  wartime  situation  assessment.  Research  and  development  is  concerned 
with  requirements  definition,  engineering  design  support  and  systems  performance 
assessment.  Test  and  evaluation  focuses  on  early  operational  assessment,  development 
and  operational  test  design;  and  operational  excursions  and  post-test  analysis.  Analysis  is 
concerned  with  campaign  analysis,  force  stmcture  assessment,  system  configuration 
determination,  sensitivity  analysis  and  cost  analysis.  Production  and  logistics  covers 
system  producibUity  assessment,  industrial  base  appraisal  and  logistics  requirements 
determination  (17:  2-2). 

The  guide  suggests  that  there  are  three  general  types  of  models:  wargaming; 
training;  and  acquisition.  “Wargaming  models  range  from  single  engagement  (one-on- 
one)  to  joint  theater  level  campaign  operations.  Training  models  range  from  single 
template  instructional  systems  to  complex  virtual  reality  simulations.  Acquisition  models 
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system  acquisition. 


Law  and  Keiton’s  Simnuiatoorii  Modeling  and  Analysis 

In  addition  to  using  both  the  DoD  Master  Plan  and  the  Systems  Acquisition 
Manager’s  Guide,  Law  and  Kelton’s  text  is  valuable.  It  discusses  the  fundamentals  of 
modeling  and  simulation  design.  This  leads  to  the  idea  that  some  of  the  potential  traits 
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could  be  tool  specific.  For  instance,  a  model  that  represents  a  system  at  a  particular  point 
in  time  is  referred  to  as  static  (11:  3).  Conversely,  models  that  represent  a  system  as  it 
evolves  over  time  are  referred  to  as  dynamic.  For  those  models  that  are  dynamic,  how  it 
represents  a  system  could  be  distinguishing  trait.  If  the  dynamic  model  evolves  over  time 
and  the  state  variables  only  change  at  a  countable  number  of  points  in  time,  then  the  model 
is  considered  discrete  (11:  3).  However,  should  the  state  variables  change  continuously 
over  time,  then  the  dynamic  model  is  considered  to  be  continuous  (11: 46).  A  model 
could  also  be  characterized  as  how  it  uses  random  variables.  A  deterministic  model  uses 
no  random  variables  whereas  a  stochastic  model  contains  one  or  more  random  variables 
(1 1: 3).  All  of  these  ideas  could  provide  very  descriptive,  definitive  traits  for  describing 
MS&A. 

Personal  Interviews 

Another  source  of  information  for  developing  potential  traits  for  the  database  was 
personal  interviews  and  conversations  with  members  of  the  Air  Force  MS&A  community. 
The  most  productive  were  those  interviews  with  Maj  Steve  ChimelsM,  the  thesis  sponsor 
from  HQ  AFMC/XRX,  conversations  with  ILt  Tim  Ragsdale  and  Capt  Bill  Greer  during  a 
visit  to  the  Space  and  Missile  Center  (SMC),  telephone  conversations  with  Mr.  Bemie 
McKinney  at  Edwards  AFB,  and  discussions  with  Mr.  Richard  J.  Simard  from  the  Rome 
Air  Development  Center  Laboratory  at  Giiffiss  AFB. 

Maj  Chimelski  recounted  many  past  iterations  with  the  goal  of  developing  some 
mechanism  to  track  the  status  of  the  AFMC  MS&A  assets.  It  is  believed  that  such  a  tool 
would  reduce  yearly  expenditures  on  developing  new  MS&A  applications  by  reducing 
redundant  applications.  Most  previous  work  centered  around  developing  paper  databases 
that  were  inconvenient  to  use,  difficult  to  maintain,  and  provided  no  search  capability  (3). 
This  led  to  the  request  for  a  database  that  would  “allow  the  users  to  access  the  titles,  short 
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descriptions,  and  POCs  of  the  individual  models  and  simulations”  (4).  Table  1  provides  a 
brief  recap  of  the  last  attempt  to  create  a  paper  database  to  fulfill  the  needs  of  the  AFMC 
MS&A  community. 

The  interviews  and  information  collected  from  the  SMC  in  Los  Angeles  AFB,  CA, 
the  412  Test  Wing  at  Edwards  AFB,  CA;  and  Rome  Air  Development  Center  Lab  at 
Griffiss  AFB,  NY  show  that  there  is  great  interest  in  having  a  database  to  track  MS&A 
tools.  All  three  of  these  organizations  have  developed  their  own  databases  and  two  of 
them  have  loaded  their  listings  onto  the  Internet.  Table  2  shows  the  data  fields  captured 
by  each  of  these  existing  databases. 

The  fact  that  some  of  the  AFMC  organizations  have  already  developed  their  own 
databases  and  that  others  are  in  the  process  of  developing  one  shows  that  the  AFMC 
MS&A  community  is  keenly  interested  in  having  a  MS&A  database.  However,  the 
interest  doesn’t  stop  at  AFMC  because  the  DoD  has  also  done  some  work  in  developing 
databases  or  catalogs  to  advertise  their  MS&A  capabilities. 
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Table  1.  Recap  of  the  Previous  AFMC  MS&A  Database  Development  Effort. 


AFMC  MODELS  AND  SIMULATIONS  SURVEY 

FIELD 

DEFINITION 

1.  Title 

Name  of  the  model  or  simulation 

2.  Model  Owner/Maintainer 

Name  of  Owner 

3.  Point  of  Contact 

Name,  Organization,  and  Phone 

4.  Purpose 

Purpose  of  the  model  or  simulation 

5.  Type:  Choose  one  of  these: 

Policy 

Theater 

Campaign 

Engagement 

Engineering/Component 

Support 

Database 

Probability  Model 

Other  (Specify) 

Categorize  by  type  of  simulation 

6.  Functional  Area  Application:  Choose  one  or 
more  of  the  following: 

Analysis 

Research  and  Development 

Education,  training,  and  Military  Operations 

Test  and  Evaluation 

Production  and  Logistics 

Manpower  and  Personnel 

Other  (Specify) 

Categorize  by  functional  area  of  application 

7.  Description 

Describe  the  model  or  simulation 

8.  Verification,  Validation,  and  Accreditation 

Actions  taken  to  obtain  verification, 
validation,  or  accreditation  on  the  model  or 
simulation 

9.  Description 

Description  of  model  or  simulation 

10.  Hardware 

Hardware  requirements  of  the  model  or 
simulation 

1 1 .  Software 

Software  requirements  of  the  model  or 
simulation 

12.  Database 

Database  requirements  of  the  model  or 
simulation 

13.  Developer 

Air  Force  (Specify  which  agency) 

Other  government 

Contractor 

Agency  which  developed  the  model  or 
simulation 

14.  Costs 

Cost  incurred  in  developing  the  model  or 
simulation 

15.  Frequency  and  Ease  of  Use 

Frequency  and  ease  of  use  of  the  model  or 
simulation 
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DoD  Efforts 


This  is  an  AFMC  effort  with  a  primary  goal  of  cataloging  AFMC  MS&A  assets. 

However,  there  are  some  DoD  efforts  which  may  impact  future  applications  of  this 

database.  First,  in  conversations  with  AF/XOM  it  appears  that  the  DoD  is  moving 

towards  a  unified  format  for  describing  the  capabilities  of  M&S  tools.  It  also  seems  likely 

that  DoD  may  soon  be  starting  a  consolidated  database  of  M&S  assets  through  out  the 

DoD.  This  effort  may  be  overcome  by  DoD  directions  to  conform  to  their  structure  and 

data  requirements.  However,  at  this  point  in  time,  no  directions  have  been  received.  The 

Defense  Simulation  &  Modeling  Office  (DMSO)  maintains  a  number  of  catalogs  on  the 

Internet.  As  of  June  95,  there  were  nine  catalogs  covering  the  four  services: 

Air  Force  Studies  and  Analysis  Agency  (AFSAA)  M&S  Catalog 
Ballistic  Missile  Defense  Organization  (BMDO)  M&S  Catalog 
J-8  M&S  Catalog 

ARMY  Models  &  Simulations:  Army  Integrated  Catalog  (MOSAIC) 

US  Air  Force  Rome  Laboratory  M&S  Catalog 
TRANSCOM  System  Model  Catalog 

Catalog  of  War  Games,  Training  Games,  and  Combat  Simulations 
Navy  Catalog  of  Models  and  Combat  Simulations 
USAF  SMC/XR 

There  are  some  drawbacks  to  these  catalogs.  First,  not  all  of  the  assets  of  each  of 
the  services  are  represented.  Second,  they  only  allow  a  key  word  search  to  determine 
applicable  models  and  this  capability  is  only  available  in  seven  of  the  nine  catalogs. 
Another  drawback  is  that  each  of  the  catalogs  varies  in  content  and  amount  of  data  given 
for  an  particular  model.  Even  considering  these  drawbacks,  the  catalogs  provide  a  service 
which  is  unavailable  from  any  other  source. 

Summary 

This  chapter  provides  some  background  information  needed  to  understand  the 
importance  and  relevance  of  this  research.  We  are  dealing  with  a  requirement  to  compile 
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and  use  data  concerning  AFMC’s  M&S  assets,  in  other  words  we  are  dealing  with  an 
information  system.  The  literature  search  reviewed  the  different  types  of  information 
systems  and  basically  identifies  our  system  as  being  a  Transaction  Processing  System 
(TPS).  Given  that  we  knew  what  kind  of  system  we  had,  we  addressed  the  requirements 
of  the  customer  in  terms  of  what  they  wanted.  This  led  to  a  determination  that  a  database 
would  be  the  most  suitable  vehicle  for  recording  the  characteristics  and  traits  of  M&S. 
This  led  to  a  discussion  of  the  steps  involved  in  developing  a  database.  However,  the 
database  is  merely  the  proof  of  concept  for  the  research  emphasis  of  this  thesis.  We  are 
interested  in  determining  the  important  traits  and  characteristics  of  MS&A  so  that  we  can 
catalog  and  search  AFMC’s  inventory.  The  first  step  of  determining  applicable  traits  was 
a  search  of  the  literature  to  see  if  any  previous  efforts  had  been  done  in  this  subject  area. 

A  few  taxonomies  exist  for  specialized  applications  of  M&S,  but  nothing  appears  to  have 
been  done  on  an  overall,  unifying  taxonomy  scheme.  Thus  we  basically  need  to  start  from 
scratch.  We  researched  several  regulations,  instractions,  and  texts  which  discussed  M&S. 
We  also  interviewed  and  visited  with  many  current  practitioners  in  the  field.  These 
sources  will  provide  the  basis  for  the  research  effort.  Finally ,  we  briefly  covered  the 
DoD’s  efforts  because  there  is  a  potential  chance  that  this  effort  may  be  overcome  by  a 
chain  of  events  outside  of  AFMC’s  control. 


28 


Hi.  Methodology 


Chapter  Overview 

This  chapter  describes  the  processes  that  will  be  followed  to  achieve  the  objectives 
of  identifying  the  different  traits  of  MS&A,  identifying  which  traits  are  most  useful,  and 
developing  a  database  based  on  those  most  useful  traits.  We  will  address  how  each 
investigative  question  will  be  answered. 


What  Traits  Differentiate  One  MS&A  Program  From  Another? 

The  first  investigative  question  addresses  the  issue  of  what  differentiates  one 
MS&A  tool  from  another.  In  order  to  answer  this  question,  a  literature  search  was 
conducted  to  provide  the  basic  foundation  of  terminology  and  usage.  The  literature  search 
was  broadened  by  interviews  with  field  and  headquarters  personnel  to  define  current  Air 
Force  aspects.  The  literature  search  and  interviews  suggested  that  there  are  many  traits 
which  could  be  useful  for  attempting  to  categorize  MS&A  applications.  These  avenues 
will  be  explored  by  identifying  which  traits  are  most  useful  in  a  database. 

Which  Traits  Are  Most  Useful  to  Analysts  in  Selecting  MS&A  Software? 

The  second  investigative  question  explores  the  usefulness  of  MS&A  traits 
identified  from  the  literature  search  and  interviews.  A  briefing  (Appendix  A)  was 
developed  and  presented  to  a  panel  of  experts  from  all  portions  of  the  AFMC  simulation 
community  to  elicit  their  ideas  and  concerns  about  this  study.  Based  upon  the  results  of 
the  literature  search,  interviews,  and  discussions,  a  survey  was  developed  to  collect  the 
AFMC  MS&A  community’s  thoughts  on  the  usefulness  of  each  potential  trait.  Part  of  the 
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survey  uses  a  Likert  scale  to  gauge  a  respondent’s  opinion  of  the  usefulness  of  each  item 
and  the  scores  are  totaled  to  measure  the  respondent’s  attitude  (5:  179).  The  survey  is 
shown  in  its  entirety  in  Appendix  B,  but  its  construction  is  discussed  here. 
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L  ITEM:  Numbers  and  groups  the  fields.  Item  also  provides  a  tracking 
mechanism  between  the  survey  and  the  source  document. 

2.  POTENTIAL  FIELD:  Name  of  the  field  and  subelements,  if  any. 

3.  FIELD  ENTRY:  How  the  field  would  appear  in  the  database.  It  could  be  a 
text  entry  or  a  coded  entry.  Where  a  field  entry  is  given,  it  is  only  a  suggestion. 
It  may  or  may  not  be  appropriate  for  the  circumstances.  In  some  instances,  no 
suggestion  is  given  and  the  subject’s  comments  and  recommendations  are 
needed. 

4.  APPLY  TO:  Some  potential  fields  may  have  application  in  all  portions  of  the 
database  (e.g.  Title)  whereas  other  fields  may  only  apply  to  one  portion  of 
the  database  (e.g.  Class  of  Simulation  may  only  apply  to  Models).  The  three 
logical  groupings  thus  far  defined  are: 

-  Models:  actual  tool  or  software. 

-  Databases:  data  files  necessary  to  run  the  models. 

-  Analyses:  studies  that  have  been  done  using  a  model. 

Any  or  all  three  of  these  may  be  appropriate  for  a  given  proposed  field. 

5.  USEFULNESS:  Ranking  of  subject’s  perception  of  the  field’s  utility: 

1  =  Extremely  Useful,  the  database  would  not  make  much  sense 

or  have  much  utility  without  it. 

2  =  Useful. 

3  =  Marginally  Useful,  the  field  is  of  intermediate  necessity. 

4  =  Minimally  Useful. 
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5  =  Not  Useful  At  All,  it’s  a  waste  of  time  and  space  to  have  such  a 
field. 

6.  EXCLUSIVE:  Subject  is  to  determine  if  the  field  can  take  on  multiple  values 
for  a  particular  item.  For  instance,  if  the  Class  of  Simulation  is  live,  can  it  be 
constructive  also?  In  some  cases,  mutual  exclusion  may  be  necessary  or 
beneficial;  in  other  cases  it  may  not. 

The  results  of  this  survey  will  provide  the  basis  for  defining  the  MS&A  database. 
The  responses  to  the  survey  will  be  analyzed  to  determine  which  potential  traits  are 
viewed  by  the  MS&A  community  as  being  most  needed  in  a  database.  Two  main  pieces 
of  information  will  be  obtained  from  the  survey; 

1.  Each  potential  field  will  be  evaluated  as  to  whether  or  not  it  should  be  included 
in  each  section  of  the  database  (i.e.  in  the  Model,  Database,  or  Analysis  section).  This  will 
be  derived  in  a  two  step  process.  First,  in  order  to  be  included,  each  field  must  yield  an 
average  value  of  2.50  or  less.  The  cutoff  value  of  2.50  was  chosen  because  it  is  the 
midpoint  between  a  rating  of  useful  (2.00)  and  marginally  useful  (3.00).  Second,  a 
majority  of  respondents  (50%  or  greater)  must  respond  that  the  field  would  be  useful  for  a 
particular  section  of  the  database.  Let’s  suppose  for  instance  that  a  potential  field  yields 
an  average  value  of  2.00  with  7  out  of  10  respondents  for  the  Models  section;  an  average 
value  of  3.00  with  6  out  of  10  respondents  for  the  Database  section;  and  an  average  value 
of  LOO  with  2  out  of  10  respondents  for  the  Analysis  section.  In  this  case,  only  the 
Models  section  would  include  the  potential  field.  The  Database  section  would  not  pass 
muster  because  it  had  an  average  greater  than  2.50  (even  though  it  had  a  60%  response 
rate)  and  the  Analysis  section  would  not  pass  muster  because  it  only  had  a  response  rate  of 
20%  (even  though  it  had  an  usefulness  rating  of  1.00). 
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2.  Each  potential  field  will  be  evaluated  as  to  whether  or  not  it  should  be 
considered  mutually  exclusive.  This  is  to  say  whether  or  not  a  potential  field  could  use 
only  one  of  the  suggested  field  entries  or  more  than  one.  The  responses  from  the  survey 
will  be  tabulated  and  a  simple  majority  (i.e.  50%  or  greater)  will  determine  whether  or  not 
the  field  should  be  mutually  exclusive. 

Once  the  results  have  been  compiled  and  tabulated,  we  can  take  steps  to  start 
developing  the  prototype  database. 

Can  a  Database  Based  on  a  Cataloging  System  Which  Uses  Such  Traits 
Provide  Retrieval  Performance  for  Users? 

The  last  investigative  question  brings  the  first  two  questions  together  into  a 
practical  application.  The  intention  is  to  create  a  prototype  database  to  determine  whether 
or  not  MS&A  tools  can  be  cataloged  and  whether  or  not  that  database  can  provide  an 
acceptable  retrieval  capability.  The  AFMC  MS&A  database  will  be  designed  according  to 
the  steps  outlined  in  Chapter  2,  Literature  Review.  This  four  step  process  starts  with 
identifying  the  requirement  formulation  and  analysis  for  the  database.  Next  we  need  to 
develop  the  conceptual  design  of  the  database  and  identify  the  relationships  between  the 
data.  After  that  is  completed,  we  develop  a  relational  model,  build  a  data  dictionary,  and 
design  the  prototype  database.  Last,  the  database’s  physical  design  is  reviewed,  but  this 
step  is  not  part  of  this  thesis  effort.  However,  an  additional  step  of  testing  the  prototype 
database  will  be  accomplished. 

Requirements  Formulation  and  Analysis:  Much  of  the  need  for  an  AFMC 
MS&A  database  is  outlined  in  Chapter  1,  Background  and  Problem  Statement.  In 
addition  to  realizing  a  potential  savings  from  reducing  duplication  of  effort,  this  database 
could  be  used  as  a  vehicle  to  advertise  the  current  capabilities  of  the  AFMC  MS&A 
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cornniunity.  It  could  also  be  used  as  a  research  tool  and  as  a  tracking  mechanism.  The 
type  of  data  collected  will  be  the  defining  characteristics  of  MS&A  software.  Some  such 

■  ts. 


proof  of  concept  as  to  the  usefulness  of  the  MS&A  traits  used  to  develop  the  prototype 


database. 


Testing:  The  last  step  of  this  effort  consists  of  testing  the  database.  After  the 
database  has  been  designed  and  partially  populated,  an  inquiry  screen  will  be  developed 
and  testing  will  begin.  The  test  will  be  conducted  by  using  potential  users  from  one  of  the 
labs  to  evaluate  the  usefulness  of  the  database.  Each  potential  user  will  be  asked  about 
their  most  recent  simulation  experience.  Based  on  that  experience,  they  will  be  asked  to 
use  the  prototype  database  to  determine  whether  or  not  the  application  they  used  is 
identified  by  their  query.  They  will  be  also  asked  to  evaluate  the  other  applications 
suggested  by  the  database  as  to  whether  or  not  they  would  be  useful  in  the  user’s 
particular  problem  area.  They  will  also  be  asked  about  how  they  view  the  utiUty  of  the 
database.  The  prototype  will  be  evaluated  according  to  three  criteria: 

-  Did  the  database  suggest  the  model  that  the  user  had  used  in  their  simulation 
study  (provided  the  model  is  in  the  prototype)? 

-  What  is  the  perceived  usefulness  of  the  remaining  suggested  models  (using  a 
Likert  scale  similar  to  one  used  in  the  survey)? 

-  How  satisfactory  was  the  prototype  (i.e.  a  utility  rating  using  a  Likert  scale)? 


Summary 

This  chapter  provides  the  outline  of  the  approach  which  will  be  taken  to  answer 
the  investigative  questions  and  to  develop  the  MS&A  database  which  will  be  used  as  proof 
of  concept. 
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However,  a  28%  (7/25)  response  rate  was  viewed  as  being  low.  The  TPIPT  was  meeting 
at  Wright-Patterson  AEB  20-21  Jul  95  and  a  second  chance  to  promote  the  survey  was 
arranged.  Following  the  presentation,  an  additional  15  surveys  were  distributed  (no 
members  were  double  counted,  i.e.  only  one  survey  was  allowed  per  respondent).  As  a 
result,  a  total  of  16  TPIPT  members  responded,  13  with  completed  surveys  and  three  with 
general  comments  but  incomplete  surveys.  This  equates  to  a  40%  response  rate  (16/40). 
Table  3  provides  the  results  of  identifying  which  traits  the  community  feels  are  most  useful 
in  describing  MS&A  software.  Appendix  C  is  a  spreadsheet  that  shows  the  individual 
responses  and  values  which  were  used  to  determine  the  relevant  fields  represented  in 
Table  3.  As  discussed  in  chapter  3,  each  potential  field  had  to  receive  an  average 
usefulness  rating  of  2.50  or  less  and  be  considered  useful  by  more  than  50%  of  the  total 
respondents.  If  the  potential  field  did  not  pass  muster  on  either  condition,  then  it  was 
eliminated  from  further  consideration.  Appendix  D  is  a  spreadsheet  that  shows  the 
individual  responses  concerning  the  mutually  exclusive  issue  for  an  potential  field.  A 
simple  majority  was  needed  to  determine  whether  or  not  a  field  is  mutually  exclusive  or 
not.  Without  exception,  all  of  the  potential  fields  were  determined  to  be  not  mutually 
exclusive.  This  means  that  for  any  potential  field  that  has  options  or  selections,  a  MS&A 
tool  could  be  described  by  one  or  more  of  the  options.  Now  that  we  know  what  fields  are 
involved,  we  began  the  process  of  creating  an  MS&A  database. 

Can  a  Database  Based  on  a  Cataloging  System  Which  Uses  Such  Traits 
Provide  Retrieval  Performance  for  Users? 


We  will  answer  this  question  by  developing  a  prototype  MS&A  database.  In 
Chapter  2,  Literature  Review,  I  outlined  a  four  step  process  for  developing  a  database; 


Requirements  Formulation  and  Analysis:  This  first  step  has  been 
discussed  in  detail  in  previous  sections  of  this  thesis.  We  covered  the  need  for  an  AFMC 
MS&A  database  in  Chapter  1,  Background  and  Problem  Statement.  We  reviewed 
management  reports,  documentation,  operating  instructions,  policy,  guidance,  and 
regulations;  observed  the  workplace;  and  conducted  interviews  with  users  and  managers. 
This  collected  information  provided  the  basis  for  a  survey  that  was  constructed  as 
described  in  Chapter  3,  Methodology  (the  actual  survey  is  contained  in  Appendix  B).  The 
results  of  the  survey  were  discussed  above  in  the  section  on  answering  the  second 
investigative  question.  Having  collected  a  basis  of  information  and  determined  the  need 
for  the  database,  we  were  ready  to  proceed  to  the  next  step. 

Conceptual  Design:  We  will  build  upon  the  data  collected  in  the  first  step  of 
the  design  process.  Our  conceptional  design  will  be  built  by  using  an  Entity-Relationship 
(ER)  model  to  format  our  data  collected  from  the  survey.  Figure  10  shows  the  results  of 
this  tool  which  represents  the  basis  for  the  conceptual  model. 

The  model  envisions  four  major  entities;  Models,  Databases,  Analyses,  and  Office. 
Models  are  capable  of  interacting  with  Databases  and  vice  versa.  This  relationship  is  a 
many-to-many  relationship  which  means  each  model  can  interact  with  many  different 
databases  and  the  reverse  is  true  also;  that  is  many  databases  can  interact  with  many 
different  models.  This  same  situation  exists  between  the  Models  to  Analyses  and 
Databases  to  Analyses  relationships.  The  relationship  shared  between  Office  and  the  other 
three  entities  is  a  one-to-many.  In  the  case  of  Models,  this  means  that  each  model  has 
only  one  Office  that  has  the  overall  responsibility  to  maintain  that  model.  However,  each 
Office  has  the  potential  of  maintaining  more  than  one  Model.  This  same  concept  holds 
true  for  the  Office  to  Databases  and  Office  to  Analyses  relationships. 
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fields  that  will  be  an  inherent  part  of  each  entity.  Appendix  E  is  the  data  dictionary  which 
contains  the  names,  length,  type  of  data,  and  definition  for  each  field  in  the  Models  and 
Office  tables  (i.e.  the  entities  have  been  directly  translated  into  tables  for  the  prototype 
database). 


information  was  available  to  populate  the  Database  and  Analysis  sections.  Second,  the 
survey  identified  new  information  requirements  for  the  Models  table  (such  as  defining  the 
class  of  simulation,  the  military  capability  rating,  and  type  of  M&S  to  name  a  few).  The 
Models  table  has  84  fields  used  to  describe  M&S  models  or  tools.  These  fields  can  be 
further  segmented  as  being  either  descriptive  or  selective.  For  instance,  the  fields  like 
Model_Title  and  Model_Descrip  could  be  viewed  as  descriptive.  However,  fields  like 
Model_Func_Area_R&D  or  Model_Weapon_Fighter  are  selective.  Selective  means  one 
can  choose  a  model  based  on  desired  characteristics.  All  of  the  selective  fields  were 
defined  to  be  Yes/No  fields  which  allows  for  multiple  selection  in  a  particular  field 
category  (such  as  AF  Hierarchy,  Functional  Area  of  Apphcation,  or  Class  of  Simulation). 
Table  4  shows  the  Models  descriptive  fields  and  Table  5  shows  the  Models  selective  fields 
arranged  by  categories. 


Table  4.  Models  Descriptive  Fields. 


MODEL.TITLE 

MODEL_ACRONYM 

MODEL_OFFICE_SYMBOL 

MODEL_DATA_VVC_DATE 

MODEL_SECURITY_LEVEL 

MODEL_VER_DATE 

MODEL_VAL_DATE 

MODEL_ACCRED_DATE 

MODEL_REUSE 

MODEL_LEGACY 

MODEL_M&S_INTEROP 

MODEL_PLATFORM 

MODEL_RUN_TIME 

MODEL_DESCRIP 


MODEL_VERSION_NUMBER 

MODEL_DUTYORG 

MODEL_COMMON 

MODEL_DATA_VVC_CODE 

MODEL_RELEASE 

MODEL_VER_AGENCY 

MODEL_VAL_AGENCY 

MODEL_ACCRED_AGENCY 

MODEL.PORTABILITY 

MODEL_FIDELITY 

MODEL_DISTRIBUTED_OPS 

MODEL_LANGUAGE 

MODEL_LIMn’S 
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MODEL_AF_HIER_SYSTEM 


SYSTEM  SEGMENT 


Unfortunately  these  new  information  requirements  were  the  foundation  for 
establishing  any  kind  of  search  capability.  Assistance  was  rendered  by  Captain  Ken 
Scribner,  Mr.  Brian  Stadler,  and  Mr.  Joe  Nalepka  from  Wright  Labs  (WL)  in  identifying 
the  appropriate  coding  for  the  a  few  select  WL  models  in  the  1993  catalog.  This  was 
necessary  because  without  defining  the  models  in  terms  of  their  search  characteristics,  it 
would  have  been  impossible  to  test  the  concept  of  the  prototype  database.  The  database 
was  populated  with  over  300  Models  and  over  80  Office  entries.  Once  the  population  of 
the  database  was  completed,  we  started  on  designing  the  questionnaire  for  the  test. 

Physical  Design:  This  step  is  concerned  with  the  operational  characteristics  of 
the  database  design  and  is  beyond  the  scope  of  this  thesis.  The  prototype  database  is  for 
proof  of  concept  as  to  the  usefulness  of  the  MS&A  traits;  not  as  a  completed,  operational 
database. 

Testing:  The  last  step  in  the  process  was  a  test  of  the  prototype  to  gain  an 
indication  of  how  well  the  prototype  performed  in  the  view  of  the  potential  users.  After 
the  database  was  designed  and  partially  populated,  testing  began.  Each  potential  user 
tried  the  prototype  to  obtain  a  list  of  models  which  may  have  been  applicable  to  their  last 
M&S  application  and  completed  a  questionnaire  (Appendix  F)  to  describe  the  results  of 
their  search.  As  part  of  the  questionnaire,  each  potential  user  was  asked  about  their  most 
recent  simulation  experience.  Based  on  that  experience,  they  were  asked  to  use  the 
prototype  database  to  determine  whether  or  not  the  application  they  used  was  identified  by 
their  query.  Additionally,  they  were  asked  to  evaluate  the  other  applications  suggested  by 
the  database  as  to  whether  or  not  they  would  be  useful  in  the  user’s  particular  problem 
area.  Last,  they  were  asked  about  how  they  viewed  the  utility  of  the  database. 
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The  prototype  was  evaluated  according  to  three  criteria: 

-  Did  the  database  suggest  the  model  that  the  user  had  used  in  their 
simulation  study  (provided  the  model  is  in  the  prototype)? 

-  What  is  the  perceived  usefulness  of  the  remaining  suggested  models? 

-  How  satisfactory  was  the  prototype? 

It  was  anticipated  that  the  questionnaire  would  be  administered  to  a  small  sample 
of  six  M&S  users.  However,  after  administering  the  questionnaire  to  three  individuals  and 
obtaining  no  usable  results,  the  test  was  terminated.  In  all  three  cases,  the  users  did  not 
obtain  any  outputs  (i.e.  the  prototype  failed  to  return  any  potential  titles).  First,  the 
models  that  the  users  had  worked  with  were  not  in  the  database,  thus  the  answer  to  the 
first  question  was  no.  The  second  question  went  unanswered  because  no  titles  were 
suggested,  thus  a  rating  on  their  usefulness  was  impossible.  Finally,  the  last  question  was 
answered  with  a  favorable  result.  The  subjects’  average  rating  for  the  prototype  was  2 
which  translates  into  a  rating  of  useful.  This  was  a  heartening  result  given  that  the 
protot5q)e  failed  to  yield  any  tangible  results.  All  three  respondents  believe  that  the 
database  could  be  a  useful  tool  once  it  is  expanded  and  updated. 

The  database  was  not  sufficiently  populated  nor  specified  to  allow  for  a  useful  test. 
Of  the  306  records  contained  in  the  Models  table,  only  53  have  been  fully  specified  to 
include  some  inputs  for  the  new  information  requirements.  This  information  must  be 
acquired  and  entered  for  all  306  records  before  profitable  testing  can  be  accomplished. 
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Summary 

This  chapter  has  covered  the  results  of  the  survey,  construction  of  the  prototype 
database,  and  the  test  results  on  the  prototype.  The  survey  did  suggest  that  there  are 
many  characteristics  that  MS&A  users  would  be  interested  in  knowing.  Based  on  these 
characteristics,  the  database  structure  was  defined  through  the  analysis  of  an  ER-Model 
and  was  translated  into  the  database  structure.  Once  the  database  structure  was  defined, 
the  database  was  populated  and  testing  was  started.  Although  the  results  of  the  test  were 
disappointing,  there  are  reasons  why  the  test  failed.  Once  these  reasons  are  addressed, 
then  a  meaningful  test  of  the  prototype  database  can  be  conducted. 
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MS&A  models,  databases,  and  analyses.  Without  doubt,  we  have  been  successful  in  our 
endeavors  of  addressing  the  first  two  investigative  questions.  However,  when  it  came  to 
the  testing  of  the  prototype  database,  we  met  with  failure.  But,  this  failure  is  not  without 
cause.  Much  of  the  data  upon  which  the  success  of  the  test  hinged  was  either  out  of  date 
or  nonexistent.  It  is  believed  that  the  MS&A  database  can  provide  the  capabilities  the 
AFMC  MS&A  community  needs;  namely  a  method  to  track  their  current  inventory  and  to 
provide  access  to  their  modeling  and  simulation  capabilities  in  definitive  terms  never 
before  available.  The  following  are  some  recommendations  or  suggestions  which  will  take 
the  database  fi:om  prototype  to  reality. 


Recommandatiemis 

1.  Populate  amd  Update  the  Prototype:  Much  of  the  information  currently 
contained  in  the  prototype  is  based  on  information  from  a  1993  AFMC  effort.  However, 
much  of  this  information  is  outdated:  some  models  are  no  longer  used  or  are  outdated  and 
other  models  that  are  currently  used  are  not  listed.  This  update  procedure  can  be 
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incorporated  with  the  populating  of  the  database  by  designing  a  database  form  which 
could  allow  for  the  direct  transfer  of  information  from  the  form  into  the  database. 

Multiple  forms  will  be  necessary  to  allow  for  updating  each  of  the  main  tables;  i.e.  Models, 
Office,  Databases,  and  Analyses.  By  automating  these  forms,  the  data  will  already  be 
appropriately  formatted  for  inclusion  into  the  database  and  will  reduce  the  level  of  effort 
associated  with  collecting  the  data  and  updating  the  database. 

2.  Develop  Forms  for  the  Query  and  Report  Functions:  The  prototype  relied 
on  the  inherent  query  capabilities  of  Access.  This  function  can  be  improved  and  made 
transparent  to  the  user  by  means  of  a  query  form  or  macro  which  will  allow  an  intuitive 
selection  of  models,  databases,  or  analyses.  This  query  form  would  provide  the  user  the 
ability  to  select  search  criteria  and  specify  and/or  relationships  by  means  of  selecting  the 
appropriate  buttons  on  the  form  and  then  the  macro  will  develop  and  run  the  query. 

3.  Select  Platform  or  Mode  of  Access:  The  MS&A  community  needs  to  decide 
which  platform  they  desire  for  the  database.  For  instance,  the  database  could  be  mounted 
on  a  mainframe  or  could  be  PC  based.  Once  this  is  decided,  the  method  of  transmission  or 
access  needs  to  be  determined.  If  the  mainframe  option  is  used,  then  the  only  reasonable 
method  of  access  would  be  real  time  because  no  user  would  wish  to  use  the  database  by 
having  to  submit  a  written  request  and  wait  for  a  response.  If  the  PC  option  is  selected, 
then  the  community  has  a  number  of  alternatives.  Updates  to  the  database  could  be  made 
periodically  and  distributed  on  floppy  disk  or  on  CD  ROM.  Or  the  database  could  be 
mounted  on  a  server  and  access  could  be  made  via  DSN,  bulletin  board,  or  Internet 
hookups.  Just  having  the  database  is  not  enough,  the  product  needs  to  be  accessible 
through  the  community  in  a  timely  fashion  to  be  of  any  significant  use. 
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4.  Select!;  am  Overall  Office  or  Orgamizatiom  of  Primary  RespomsiWlllty  (OPE): 
Someone  needs  to  be  primarily  responsible  for  the  maintenance  and  oversight  of  this 
command  wide  database.  Administrative  functions  and  improvements  will  need  to  be 
done  on  a  recurring  basis  and  someone  will  need  to  do  them.  The  Modehng  and 
Simulation  Technical  Planning  Integrated  Product  Team  (M&S  TPIPT)  would  be  a  likely 
candidate.  They  are  already  active  with  membership  throughout  the  command  and  are 
responsible  for  M&S  issues  within  AFMC.  This  database  could  contribute  to  their 
oversight  and  planning  functions.  Updates  and  changes  to  the  database  could  be 
discussed,  approved,  and  implemented  in  conjunction  with  their  periodic  meetings. 

5.  EstebMsh  am  Archive  for  Models,  Databases,  amd  Analyses:  Not  all  models, 
databases,  or  analyses  will  remain  current.  Over  time  they  will  become  outdated  or 
superseded.  However,  that  is  not  to  say  that  a  review  of  past  findings,  studies,  models, 
etc.  may  not  be  beneficial.  At  this  point  in  time  each  organization  has  the  responsibility  for 
maintaining  their  own  products.  What  is  suggested  is  that  an  archive  be  developed,  so 
that  when  a  model  is  identified  as  being  outdated  it  can  be  forwarded  to  the  archive  for 
potential  future  research  or  use. 


Symmary 

This  chapter  has  addressed  the  conclusion  of  this  thesis  and  suggested  five  follow 
on  activities  or  issues  which  need  to  be  addressed.  Participants  in  this  research  showed 
great  support  for  creation  of  a  comprehensive  taxonomy  and  subsequent  development  of  a 
database  system  for  Modeling,  simulation,  and  Analysis  products.  Such  a  database  has 
great  potential  for  identifying  redundant  development  efforts,  helping  AFMC  maintain  its 
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MS&A  inventory,  and  providing  a  vehicle  for  the  crossflow  of  information  both  within  and 
outside  AFMC. 
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MS&A  DATABASE 
PROBLEM  STATEMENT 


•  AF  Expenditure  of  $60  -70  million/year 

•  Duplication  of  MS&A  functions  (50-60%) 

•  Diverse  applications  of  MS&A 

•  Numerous,  independent  locations 

•  No  centralized  catalogue  of  MS&A 

•  Little  coordination  of  development  efforts 


MS&A  DATABASE 
STUDY  GOALS 

•  Identify  useful  characteristics  and  traits 

•  Design  a  classification  system  for  MS&A 

•  Identify  user  needs  &  operating  parameters 

•  Develop  a  MS&A  database 

•  Develop  a  retrieval  protocol  to  identify 
specific  MS&A 
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Functional 


MS&A  DATABASE 
END  GOAL 


•  Program  Manager:  “  I  want  to  find  any 

MS&A  dealing  with _ ” 

-  F-16  performance 

-  probability  models 

-  risk  analysis 

-  weapons  performance 

•  And  the  database  would  yield  a  concise 
list  of  MS&A  possibilities. 


MS&A  DATABASE 
OTHER  CONSIDERATIONS 

•  Use  as  info  only  or  decision  making 

•  Collect  cost  data 

•  PC  or  mainframe  based 

•  Distributed  or  centralized 

•  Other 


survey,  circle  your  responses,  and  provide  the  results  to  me  at  end  of  the 
conference.  1  will  be  at  your  disposal  to  answer  any  questions  during  Thursday 
morning  or  all  day  Friday.  Thank  you  for  your  assistance. 

TIMOTHY  J.  WAGNER,  Capt,  USAF 
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MS&A  DATABASE  SURVEY  INSTRUCTIONS 


The  MS&A  database  is  envisioned  to  have  three  main  sections:  models, 
databases,  and  analysis.  Models  are  the  actual  programs  or  tools  used  by 
analyst  to  answer  questions  of  interest.  Examples  are  Thunder  or  TAG  Brawler. 
Databases  are  the  data  files  needed  to  run  the  models.  In  fact,  some  data  files 
can  be  used  interchangeably  or  provide  input  parameters  in  different  models. 
Analysis  is  the  study  output  addressing  a  particular  problem  or  issue  that  uses  a 
MS&A  model  and/or  database.  Analysis  can  consist  of  written  reports, 
spreadsheets,  or  other  like  items. 

For  each  potential  field,  you  will  be  asked  to  do  the  following: 

1 .  Suggest  a  field  entry  for  some  potential  fields. 

2.  Identify  the  applicability  of  the  potential  field  to  models  (M),  databases 
(D),  and/or  analysis  (A).  Any  or  all  may  be  circled.  For  example,  “Title”  would 
likely  have  all  three  circled  while  “Class  of  Simulation”  may  only  have  “M” 
circled. 

3.  Rank  the  usefulness  of  each  potential  field  as  to  its  utility  in  an  MS&A 
database.  A  ranking  of  1  means  that  the  potential  field  would  be  a  needed  and 
integral  part  of  the  database.  Conversely,  a  ranking  of  5  means  that  the  field 
would  be  unnecessary.  Only  one  number  should  be  circled. 

4.  Identify  whether  or  not  the  field  entry  codes  would  be  mutually 
exclusive  or  more  than  one  field  entry  code  could  apply  to  a  potential  field. 

The  survey  has  six  main  columns: 

ITEM:  Numbers  and  groups  the  fields.  Item  also  provides  a  tracking 
mechanism  between  the  survey  and  the  source  document. 

POTENTIAL  FIELD:  Name  of  the  field  and  subelements,  if  any. 

FIELD  ENTRY:  How  the  field  would  appear  in  the  database.  It  could  be  a 
text  entry  or  a  coded  entry.  Where  a  field  entry  is  given,  it  is  only  a  suggestion. 


55 


It  may  or  may  not  be  appropriate  for  the  circumstances.  In  some  instances,  no 
suggestion  is  given  and  your  comments  and  recommendations  are  needed. 

APPLY  TO:  Some  potential  fields  may  have  application  in  all  portions  of 
the  database  (such  as  Title)  whereas  other  fields  may  only  apply  to  one  portion 
of  the  database  (such  as  Class  of  Simulation  may  only  apply  to  Models 
perhaps).  The  three  logical  groupings  thus  far  defined  are: 

-  Models:  actual  tool  or  software. 

-  Databases:  data  files  necessary  to  run  the  models. 

-  Analysis:  studies  that  have  been  done  using  a  model. 

Any  or  all  three  of  these  may  be  appropriate  for  a  given  proposed  field. 

USEFULNESS:  This  is  a  ranking  of  how  useful  you  view  a  field: 

1  =  Extremely  Useful,  the  database  would  not  make  much  sense 

or  have  much  utility  without  it. 

2  =  Useful. 

3  =  Marginally  Useful,  the  field  is  of  intermediate  necessity. 

4  =  Minimally  Useful. 

5  =  Not  Useful  At  All,  its  a  waste  of  time  and  space  to  have  such  a 

field. 

EXCLUSIVE:  This  column  is  basically  asking  whether  or  not  the  field  is 
mutually  exclusive.  For  instance,  if  the  Class  of  Simulation  is  live,  can  it  be 
constructive  also?  In  some  cases,  mutual  exclusion  may  be  necessary  or 
beneficial;  in  other  cases  it  may  not. 

EXAMPLE: 


ITEM 

POTENTIAL  FIELD 

FIELD  ENTRY 

APPLY  TO 

USEFULNESS 

-A - 

EXCLUSIVE? 

1 

Title 

Text 

-©-©  0^ 

d)  2  3  4  5 

Yes  Ljvio^ 

A  Title  would  be  used  in  each  portion  of  the  database  for  naming  the 
models,  databases,  and  studies.  The  usefulness  of  such  a  field  is  extremely 
important  and  the  titles  do  not  necessarily  have  to  be  mutually  exclusive. 
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Your  participation  in  this  survey  is  voluntary.  Your  name  and  responses 
will  be  held  strictly  confidential.  Should  you  have  any  questions,  comments, 
suggestions,  or  recommendations  about  a  particular  field,  just  jot  down  your 
thoughts  on  a  separate  piece  of  paper  if  there  isn’t  enough  room  on  the  survey. 

If  you  have  any  additions,  please  submit  those  also.  When  you  are  finished  with 
your  survey,  please  give  it  to  me  at  the  end  of  the  conference.  Should  you  wish 
to  provide  any  other  information  after  the  completion  of  the  conference,  my 
phone  number  is  DSN  785-7777  ext.  2217,  my  E-mail  address  is 
twagner@afit.af.mil.,  and  my  fax  number  is  DSN  986-7988  or  commercial  513- 
476-7988.  Thank  you  for  your  assistance. 

TIMOTHY  J.  WAGNER,  Capt,  USAF 
GCA-95S 
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ITEM  I  POTENTIAL  FIELD  I  FIELD  ENTRY  I  APPLY  TO  I  USEFULNESS  I  EXCLUSIVE 
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M  &  S  Interoperabilit' 


ITEM  I  POTENTIAL  FIELD  I  FIELD  ENTRY  I  APPLY  TO  1  USEFULNESS  I  EXCLUSIVE 


bi 


ITEM  I  POTENTIAL  FIELD  I  FIELD  ENTRY 
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item  I  POTENTIAL  FIELD  I  FIELD  ENTRY _ |  APPLY  TO  | _ USEFULNESS  |  EXCLUSIVE 


EXPLANATION  OF  SURVEY  ITEMS 


SOURCES.  Each  citation  is  taken,  verbatim,  from  the  referenced  sources  below. 
After  each  citation,  the  source  (i.e.  A,  B,  C,  or  D)  and  the  associated  page 
reference  is  given  in  parenthesis: 

A.  Under  Secretary  of  Defense  (Acquisition  and  Technology),  DoD 
5000.59-Paa.  Modeling  and  Simulation  (M&S)  Master  Plan 
(Draft).  January  1 995. 

B.  Colonel  Lalit  K  Piplani,  Lt  Colonel  Joseph  G.  Mercer,  and  Lt  Colonel 
Richard  O.  Roop,  Systems  Acquisition  Manager’s  Guide  for  the  Use  of 
Models  and  Simulation,  September  1 994. 

C.  Averill  M.  Law  and  W.  David  Kelton,  Simulation  Modeling  and 
Analysis.  c1982. 

D.  Ragsdale,  Tim  1  Lt  and  Greer,  William  Capt.  SMC/XRES,  Los  Angles 
AFB,  CA.  Personal  Interview,  20  -  24  Mar  1995. 


CITATIONS  (ordered  as  they  appear  in  the  survey): 

1 .  TITLE.  The  title  of  the  MS&A  tool,  database,  or  analysis.  (D). 

2.  ACRONYM.  The  acronym  associated  with  the  title  of  the  MS&A  tool, 
database,  or  analysis,  if  any.  (D). 

3.  COMMON-USE  M&S.  M&S  applications,  services,  or  materials 
provided  by  a  DoD  Component  to  two  or  more  DoD  Components.  (DoDD 
5000.59)  (A,  page  ix). 

4.  COMPLEX  DATA.  Data  that  cannot  be  characterized  as  a  single 
concept,  atomic  data  element  as  defined  in  DoD  8320.1  -M-1 .  Complex  data 
includes  most  scientific  and  technical  data.  It  has  been  recently  categorized  by 
the  Complex  Data  Task  Force  into  (a)  highly  derived  data  (e.g.,  probability 
hit/kill):  (b)  objects  utilizing  the  concepts  of  multiple  inheritance  (e.g.,  student- 
assistant  is  subclass  of  student  class  and  employee  class),  multiple  root 
hierarchies  (e.g.,  a  tank  is  a  vehicle  and  a  tank  is  a  weapon  where  “vehicle  and 
“weapon”  are  each  roots),  and  polymorphic  attributes  (e.g.,  “capacity”  for 
different  types  of  aircraft  may  mean  number  of  people,  pounds  of  cargo,  or 
gallons  of  fuel);  (c)  compositions  such  as  command  hierarchies,  road  networks, 
images  (binary  large  objects  (BLOBS)),  compound  documents:  and  (d)  artifacts 
of  legacy  systems  and  physical  constraints  (e.g.,  aircraft  category  and  mission  in 
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one  data  element,  intelligence  facility  code  where  the  first  few  bytes  define  how 
the  rest  of  the  field  is  used).  (Eight  l/DBTWG  Conference.  July  1994)  (A,  page 
ix). 


5.  DATA  VERIFICATION,  VALIDATION,  &  CERTIFICATION  (VV&C). 

The  process  of  verifying  the  internal  consistency  and  correctness  of  data, 
validating  that  it  represents  real  world  entities  appropriate  for  its  intended 
purpose  or  an  expected  range  of  purposes,  and  certifying  it  as  having  a 
specified  level  of  quality  or  as  being  appropriate  for  a  specified  use,  type  of  use, 
or  range  of  uses.  The  process  has  two  perspectives:  producer  and  user 
process.  (A,  page  x). 

a.  DATA  VERIFICATION.  Data  producer  verification  is  the  use  of 
techniques  and  procedures  to  ensure  that  data  meets  constraints  defined  by 
data  standards  and  business  rules  derived  from  process  and  data  modeling. 

Data  user  verification  is  the  use  of  techniques  and  procedures  to  ensure  that 
data  meets  user  specified  constraints  defined  by  data  standards  and  business 
rules  derived  from  process  and  data  modeling,  and  that  data  are  transformed 
and  formatted  properly.  (Eight  l/DBTWG  Conference,  July  1994).  (A,  page  xi). 

b.  DATA  VALIDATION.  The  documented  assessment  of  data  by 
subject  area  experts  and  its  comparison  to  know  or  best-estimate  values.  Data 
user  validation  is  that  documented  assessment  of  data  as  appropriate  for  use  in 
an  intended  model.  Data  producer  validation  is  that  documented  assessment 
within  stated  criteria  and  assumptions.  (Eight  l/DBTWG  Conference,  July  1994). 
(A,  page  xi). 

c.  DATA  CERTIFICATION.  The  determination  that  data  have 
been  verified  and  validated.  Data  user  certification  is  the  determination  by  the 
application  sponsor  or  designated  agent  that  data  have  been  verified  and 
validated  as  appropriate  for  the  specific  M&S  usage.  Data  producer  certification 
is  the  determination  by  the  data  producer  that  data  have  been  verified  and 
validated  against  documented  standards  or  criteria.  (Eight  l/DBTWG 
Conference,  July  1994).  (A,  page  x). 

6.  CLASS  OF  SIMULATION.  The  categorization  of  simulation  into  live, 
virtual,  and  constructive  is  problematic,  because  there  is  no  clear  division 
between  these  categories.  The  degree  of  human  participation  in  the  simulation 
is  infinitely  variable,  as  is  the  degree  of  equipment  realism.  This  categorization 
also  suffers  by  excluding  a  category  for  simulated  people  working  real 
equipment  (e.g.,  smart  vehicles).  (A,  page  xiii). 

a.  LIVE  SIMULATION.  A  simulation  involving  real  people 
operating  real  systems.  The  categorization  of  simulation  into  live,  virtual,  and 
constructive  is  problematic,  because  there  is  no  clear  division  between  these 
categories.  The  degree  of  human  participation  in  the  simulation  is  infinitely 
variable,  as  is  the  degree  of  human  participation  in  the  simulation  is  infinitely 
variable,  as  is  the  degree  of  equipment  realism.  The  categorization  also  suffers 
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by  excluding  a  category  for  simulated  people  working  real  equipment  (e.g.  smart 
vehicles).  (A,  page  xiii). 

b.  VIRTUAL  SIMULATION.  A  simulation  involving  real  people 
operating  simulated  systems.  Virtual  simulations  inject  human-in-the-loop  (HITL) 
in  a  central  role  by  exercising  motor  control  skills  (e.g.,  flying  an  airplane), 
decision  skills  (e.g.,  committing  fire  control  resources  to  action),  or 
communication  skills  (e.g.,  as  members  of  a  C41  team).  (A,  page  xiii). 

b.(1).  HUMAN-IN-THE  LOOP.  Virtual  simulation  brings  the 
system  (or  subsystem)  and  its  operator  together  in  a  synthetic,  or  simulated 
environment.  Although  this  document  uses  the  term  human-in-the-loop  to 
represent  these  simulations,  other  names  include  man-in-the-loop,  warfighter-in- 
the-loop,  or  person-in-the-loop.  (B,  page  4-3). 

b.(2).  VIRTUAL  PROTOTYPES.  A  more  advanced  concept 
for  virtual  simulation  is  on  our  doorstep-  virtual  prototyping.  In  this  realm,  a 
three-dimensional  electronic,  virtual  mockup,  of  system  or  subsystem  allows  an 
individual  to  interface  with  a  realistic  computer  simulation  within  a  synthetic 
environment.  (B,  page  4-3). 

c.  CONSTRUCTIVE  MODEL  OR  SIMULATION.  Models  and 
simulations  that  involve  simulated  people  operating  simulated  systems.  (A,  page 
xiii). 


7.  DATA  QUALITY.  The  correctness,  timeliness,  accuracy, 
completeness,  relevance,  and  accessibility  that  make  data  appropriate  for  use. 
(Defense  Data  Repository  System  (DDRS)  end-user  manual,  24  August  1992) 
Quality  statements  are  required  for  source,  accuracy  (positional  and  attribute), 
up-to-dateness/currency,  logical  consistency,  completeness  (feature  and 
attribute),  clipping  indicator,  security  classification,  and  releasability.  (The 
Digital  Geographic  Information  Exchange  Standard  (DIGEST),  Edition  1.2. 
January  1994).  (A,  page  x). 

8.  MODEL  VERIFICATION,  VALIDATION,  AND  ACCREDITATION 
(VV&A). 

a.  ACCREDITATION.  The  official  certification  that  a  model  or 
simulation  is  acceptable  for  use  for  a  specific  purpose.  (DoDD  5000.59).  (A, 
page  viii). 

b.  VALIDATION.  The  process  of  determining  the  extent  to  which  a 
model  or  simulation  is  an  accurate  representation  of  the  real  world  from  the 
perspective  of  the  intended  use(s)  of  the  model  or  simulation.  (DoDD  5000.59). 
(A,  page  xvi). 

c.  VERIFICATION.  The  process  of  determining  that  a  model  or 
simulation  implementation  accurately  represents  the  developer’s  conceptual 
description  and  specification.  Verification  also  evaluates  the  extent  to  which  the 
model  or  simulation  has  been  developed  using  sound  and  established  software 
engineering  techniques.  (DoDD  5000.59).  (A,  page  xvii). 
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9.  TECHNICAL  GOALS  FOR  THE  HIGH-LEVEL  ARCHITECTURE.  A  set 
of  ten  technical  goals  for  the  high-level  simulation  architecture,  corresponding  to 
general  capabilities  desired  for  simulation  systems,  is  as  follows:  (A,  page  A-2). 

a.  Entity-Level  Representation.  The  simulation  represents  entities 
that  are  appropriate  to  observation  by  the  intended  end  user. 

b.  Interoperability.  Appropriate  simulations  and  C4I  systems 
operate  in  concert  by  exchanging  information  with  one  another. 

c.  Reuse.  Components  of  one  simulation  can  be  used  in  another 
appropriate  simulation. 

d.  Portability.  The  simulation  can  be  run  on  a  variety  of  computing 

platforms. 

e.  Distributed  Operation.  Operation  of  the  simulation  can  be 
spread  across  several  platforms,  if  need  be,  particularly  to  collocate  simulation 
assets  with  geographically  dispersed  users. 

f.  Legacy  Interface.  New  simulations  will  interoperate  with  some 
selected  set  of  existing  simulations. 

g.  Scalability.  The  architecture  for  the  simulation  allows 
appropriate  growlh  in  the  number  of  entities  accommodated,  their  types,  and 
their  level  of  resolution. 

h.  Broad  Functional  Applicability.  The  architecture  developed  for 
simulations  for  one  functional  purpose  (e.g.  training)  is  extendible  for  other 
functional  purposes  (e.g.,  analysis),  where  appropriate. 

i.  Technological  Evolvability.  The  architecture  for  simulation 
allows  new  technologies  to  be  used  in  the  simulation  as  they  become  available. 

j.  COTS/GOTS  Use.  The  architecture  enables  maximum  feasible 
use  of  commercial  off-the-shelf  (COTS)  and  government  off-the-shelf  (GOTS) 
products. 

10.  AGGREGATION.  The  ability  to  group  entities  while  preserving  the 
effects  of  entity  behavior  and  interaction  while  grouped.  (Proceedings  of 
Conference  on  Variable-Resolution  Modeling,  Washington,  DC,  Ed.  Paul  Davis 
and  Richard  Hillestad,  May  1992).  (A,  page  viii). 

1 1 .  DISAGGREGATION.  The  ability  to  represent  the  behavior  of  an 
aggregated  unit  in  terms  of  its  component  entities.  If  the  aggregate 
representation  did  not  maintain  state  representations  of  the  individual  entities, 
then  the  decomposition  into  the  entities  can  only  be  notional.  (A,  page  xi). 

12.  DISTRIBUTED  INTERACTIVE  SIMULATION  (DIS).  (1)  Program  to 
electronically  link  organizations  operating  in  the  four  domains:  advanced 
concepts  and  requirements;  military  operations;  research,  development,  and 
acquisition;  and  training.  (A,  page  xii). 

13.  FIDELITY.  The  accuracy  of  the  representation  when  compared  to  the 
real-world.  (A,  page  xii). 
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14.  M  &  S  INTEROPERABILITY.  The  ability  of  a  model  or  simulation  to 
provide  services  to,  and  accept  services  from,  other  models  and  simulations, 
and  to  use  the  services  so  exchanged  to  enable  them  to  operate  effectively 
together.  (DoDD  5000.59).  (A,  page  xv). 

15.  RESOLUTION.  The  degree  of  detail  and  precision  used  in  the 
representation  of  real-world  aspects  in  a  model  or  simulation;  granularity.  (A, 
page  xvi). 

16.  MILITARY  CAPABILITY.  Future  M&S  Support  to  the  Four  Pillars  of 
Military  Capability.  M&S  can  substantially  improve  capability  and  decision 
making  in  each  of  the  four  pillars  of  military  capability:  (1)  readiness,  (2) 
modernization,  (3)  force  structure,  and  (4)  substainability.  There  are  very 
challenging  aspects  to  these  descriptions,  and  achieving  full  capabilities  will 
require  long-term,  systematic,  coordinated  efforts  across  DoD.  (A,  page  2-3, 
para  C.). 

17.  SCALABILTY.  The  ability  of  a  distributed  simulation  to  maintain  time 
and  spatial  consistency  as  the  number  of  entities  and  accompanying  interactions 
increase.  (The  DIS  Vision,  Version  1,  May  1994).  (A,  page  xvi). 

18.  FUNCTIONAL  AREA  OF  APPLICATION.  ...It  includes  all  types  of 
models  and  simulations  and  embraces  the  full  range  of  M&S  interaction  between 
the  scope  of  the  simulation,  sponsoring  component  objectives  and  functional 
area  requirements  (e.g.  education,  training  and  military  operations;  analysis; 
research  and  development;  test  and  evaluation;  production  and  logistics).  ...to 
conduct  research,  development,  test  and  evaluation  activities  while  also  using 
advanced  simulations  for  design,  manufacturing,  and  logistical  support  functions. 
(A,  page  2-2,  para  B.1 .  and  B.2.). 

ALSO:  The  user  community  is  divided  into  the  following  functional  areas: 
research  and  development:  test  and  evaluation;  analysis  and  production  and 
logistics.  Specific  applications  for  each  of  the  functional  areas  are  broken  out 
below. 

Education,  training  and  operations.  Re-creation  of  historical  battles, 
doctrine  and  tactics  development,  command  and  unit  training,  operational 
planning  and  rehearsal,  and  wartime  situation  assessment. 

Research  and  development.  Requirements  definition,  engineering  design 
support  and  systems  performance  assessment. 

Test  and  evaluation.  Early  operational  assessment,  development  and 
operational  test  design;  and  operational  excursions  and  post-test  analysis. 

Analysis.  Campaign  analysis,  force  structure  assessment,  system 
configuration  determination,  sensitivity  analysis  and  cost  analysis. 
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Production  and  logistics.  System  producibility  assessment,  industrial 
base  appraisal  and  logistics  requirements  determination.  (B,  page  2-2,  para 
2.1.). 

19.  ADVANCED  DISTRIBUTED  OPERATION  (ADS).  The  ADS  is  an 
emerging  form  of  simulation  that  has  demonstrated  the  ability  to  link  different 
types  of  simulators  at  dispersed  locations;  permitting  the  simulators  and  their 
crews  to  conduct  operations  on  the  same  simulated  battlefield  environment.  (B, 
page  4-13,  para  4.5.4.). 

20.  M&S  DIMENSIONS.  Taken  from  Figure  2-1 ,  page  2-3.  This  figure 
depicts  a  cube  with  three  faces  showing.  On  the  front  face  is  the  Scope 
dimension  consisting  of  four  levels;  Theater/Campaign,  Mission/Battle, 
System/Engagement,  and  Subsystem/Component.  On  the  side  face  is  the 
Functional  Area  dimension  consisting  of  five  elements;  ETMO,  Analysis,  R&D, 
T&E,  and  P&L.  On  the  top  face  is  the  Sponsoring  Component  dimension 
consisting  of  six  entities;  Army,  Navy,  Air  Force,  Marine  Corps,  Combatant 
Commands,  and  Other.  (A,  page  2-3). 

21 .  TYPE  OF  M&S.  The  three  general  types  of  models  are:  wargaming; 
training;  and  acquisition.  Wargaming  models  range  from  single  engagement 
(one-on-one)  to  joint  theater  level  campaign  operations.  Training  models  range 
from  single  template  instructional  systems  to  complex  virtual  reality  simulations. 
Acquisition  models  range  from  physical  level  phenomenon  models  through 
engineering  component  design  tools  to  models  of  systems-in-the-end-use- 
environment.  (B,  page  1-3,  para  1.3.). 

22.  SYSTEM  ACQUISITION  PROCESS.  The  DoDD  5000.1  establishes 
broad  policies  governing  defense  systems  acquisition  programs.  It  states  that 
the  three  decision-making  support  systems  must  interact  and  interface  with  each 
other  in  order  for  the  process  to  work  effectively.  The  three  systems  illustrated 
in  Figure  2-1  are:  1)  requirements  generation,  2)  acquisition  management  and  3) 
planning,  programming  and  budgeting  system  (PPBS).  (B,  page  2-3,  para  2.2.). 

23.  AF  HIERARCHY.  The  Air  Force  uses  MS&A  at  five  different  levels: 

(1 )  Strategic/Nationai  Military  Strategy  level 

(2)  Theater/  Campaign  level 

(3)  Mission  level 

(4)  Engagement/  Submission  level 

(5)  System/ subsystem  component  (engineering)  level  (B,  page  3-16,  para 
3.6.2.). 


24.  HIERARCHY.  The  levels  within  this  hierarchy  (Army)  include: 
Engineering:  for  design,  cost,  manufacturing  and  supportability.  Provides 
measures  of  performance  (MOP). 
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Engagement:  for  evaluating  system  effectiveness  against  enemy  systems. 
Provides  measures  of  effectiveness  (MOE)  at  the  system-on-system  level. 

Mission/Battle:  effectiveness  of  a  force  package,  or  multiple  platforms 
performing  a  specific  mission.  Provides  MOE  at  the  force-on  -force  level. 

Theater/Campaign:  outcomes  of  joint/combined  forces  in  a 
theater/campaign  level  conflict,  sometimes  called  measures  of  outcome  (MOO). 
(B,  page  4-6,  para  4.4.). 

25.  STATIC.  A  static  simulation  model  is  a  representation  of  a  system  at 
a  particular  time.  (C,  page  3). 

26.  DYNAMIC.  A  dynamic  simulation  model  is  a  representation  of  a 
system  as  it  evolves  over  time.  (C,  page  3). 

27.  DETERMINISTIC.  A  simulation  model  is  said  to  be  deterministic  if  it 
contains  no  random  variables.  (C,  page  3). 

28.  STOCHASTIC.  A  simulation  model  is  stochastic  if  it  contains  one  or 
more  random  variables.  (C,  page  3). 

29.  DISCRETE.  Discrete-event  simulation  concerns  the  modeling  of  a 
system  as  it  evolves  overtime  by  a  representation  in  which  the  state  variables 
change  only  at  a  countable  number  of  points  in  time.  (C,  page  4). 

30.  CONTINUOUS.  Continuous  simulation  concerns  the  modeling  over 
time  of  a  system  by  a  representation  in  which  the  state  variables  change 
continuously  with  respect  to  time.  (C,  page  46). 

31.  TYPES  OF  PLATFORMS.  Main  frames,  personal  computer,  or 
workstations  as  some  examples.  (D). 

32.  LANGUAGE.  FORTRAN,  COBOL,  Pascal,  BASIC,  etc.  (D). 

33.  RUN  TIME.  How  long  it  takes  to  run  the  simulation;  minutes,  hours, 
days,  or  weeks.  (D). 

34.  ROLES  OF  AEROSPACE  POWER.  A  potential  discriminator  of  the 
various  types  of  MS&A  which  could  be  useful  for  organization  metrics.  (D,  also 
from  AFM  1-1,  Vol  I). 

35.  JOINT  M&S.  Representations  of  joint  and  Service  forces, 
capabilities,  equipment,  materiel,  and  services  used  in  the  joint  environment  or 
by  two,  or  more.  Military  Services.  (DoDD  5000.59).  (A,  page  xiii). 
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36.  WEAPON  SYSTEM.  Fighters,  tankers,  bombers,  satellites,  missiles, 
communications,  intelligence,  computers,  etc.  (D). 

37.  SYSTEM  SEGMENT.  An  added  discriminator  for  the  “Weapon 
System”  category.  For  instance,  a  user  might  be  interested  in  fighter 
communication  systems  or  fighter  missiles.  This  field  would  include  descriptors 
such  as  avionics,  navigation,  communications,  weapons,  missiles,  AGE, 
maintenance  support,  cost,  etc.  (D). 

38.  LIMITATIONS.  A  text  entry  on  describing  limiting  factors  that  affects 
the  operation  of  the  program.  (D). 

39.  FREQUENCY  OF  USE.  How  often  the  item  is  used;  daily,  weekly, 
monthly,  or  yearly.  (D). 

40.  OPR.  The  person  to  contact  to  obtain  more  information.  It  would 
include  name,  rank,  and  phone  number  at  a  minimum.  (D). 

41 .  DESCRIPTION.  A  text  entry  that  describes  the  program  and 
highlights  any  additional  information  that  the  OPR  feels  is  important  and  not 
contained  in  other  fields.  (D). 

42.  DOMAIN.  Air,  land,  sea,  or  any  combination.  (D). 
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APPENDIX  C 


SURVEY  RESULTS  CONCERNING  USEFUL  TRAITS 


This  appendix  covers  the  results  of  the  MS&A  database  survey.  The  spreadsheet, 
which  follows,  contains  21  columns.  This  page  describes  the  information  contained  in 
each  column. 

Column  1,  Item:  This  column  is  a  numbering  system. 

Column  2,  Potential  Field:  This  column  is  the  title  of  the  potential  field. 

Column  3,  Apply:  This  column  breaks  out  three  potential  application  areas  for 
each  potential  field;  namely  does  it  apply  to  Models,  Databases,  or  Analyses? 

Column  4-16, 1-13:  These  columns  provide  the  numerical  responses  of  the 
survey  participants. 

Column  17,  Total:  This  column  yields  the  summed  total  across  columns  4-16. 

Column  18,  #:  This  column  shows  the  number  of  respondents  who  provide  a 
numerical  answer  to  each  potential  field. 

Column  19,  Avg:  This  column  takes  the  Total  and  divides  by  the  #  (Column 
17/Column  18)  to  yield  an  average  result  among  the  participants.  Recall  that  this  average 
must  be  2.50  or  less  in  order  for  the  potential  field  to  be  considered  for  the  database. 

Column  20,  Resp:  This  column  shows  the  number  of  respondents  to  each 
potential  field.  This  column  differs  from  Column  18  by  virtue  that  not  all  participants 
provided  a  numerical  answer  for  the  usefulness  of  a  potential  field.  In  those  cases  where  a 
respondent  identified  an  apply  answer  but  no  corresponding  usefulness  value,  then  the 
response  was  recorded  as  a  alpha  character  and  was  not  counted  for  the  averaging 
purposes  of  Column  19. 

Column  21,  %:  This  column  provides  the  percentage  of  the  number  of 
respondents  who  responded  to  each  potential  field.  Recall  that  this  percentage  must 
exceed  50%  in  order  for  a  potential  field  to  be  considered  for  the  database. 
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APPENDIX  D 


SURVEY  RESULTS  CONCERNING  MUTUAL  EXCLUSIVENESS 


This  appendix  covers  the  results  of  the  MS&A  database  survey.  The  spreadsheet, 
which  follows,  contains  19  columns.  This  page  describes  the  information  contained  in 
each  column. 

Column  1,  Item:  This  column  is  a  numbering  system. 

Column  2,  Potential  Field:  This  column  is  the  title  of  the  potential  field. 

Column  3,  Apply:  This  column  breaks  out  an  affirmative  or  negative  answer 
concerning  mutual  exclusiveness  for  each  potential  field. 

Column  4-16, 1-13:  These  columns  provide  the  responses  of  the  survey 
participants.  The  number  one  is  entered  in  the  appropriate  row  (yes  or  no)  for  each 
potential  field  to  reflect  an  answer. 

Column  17,  Total:  This  column  yields  the  summed  total  across  columns  4  - 16. 

Column  18,  #:  This  column  shows  the  number  of  respondents  who  provide  a 
answer  to  each  potential  field. 

Column  19,  Avg:  This  column  takes  the  Total  and  divides  by  the  #  (Column 
17/Column  18)  to  yield  an  average  result  among  the  participants.  Recall  that  this  average 
must  be  greater  than  50%  in  order  for  the  potential  field  to  be  considered  mutually 
exclusive. 
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ITEM  POTENTIAL  FIELD  lAPP 


Distributive  Operation  |  Yes 
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APPEMDil  E 

DATA  DiCTIOMARY 

This  data  dictionary  contains  the  definition  of  the  fields  that  comprise  the  MS&A 
prototype  database.  There  are  four  tables  which  make  up  this  database: 

-  Models 

-  Databases 

-  Analyses 

-  Office 

Of  the  four  identified  tables,  only  two  have  been  developed  for  the  prototype; 
Models  and  Office.  Below  is  the  definition  of  each  field  contained  in  the  prototype.  The 
table,  to  which  each  field  belongs,  makes  up  a  portion  of  the  title  for  that  field.  For 
instance,  the  field  entitled  ModeLAcronym  is  associated  with  the  Models  table.  Each 
entry  contains  the  name  of  the  field  and  associated  definition,  the  name  of  the  field  as  used 
in  the  database,  length  of  the  field,  type  of  data  that  is  entered  into  the  field,  and  any  notes 
concerning  the  field  for  the  Models  and  Office  tables. 


ACRONYM:  This  field  is  the  acronym  associated  with  the  title  of  the  MS&A  tool,  if  any. 

NAME:  MODEL_ACRONYM 

LENGTH:  20 

TYPE:  Text 

NOTES:  This  field,  in  conjunction  with  MODEL_TITLE  and 

MODEL_VERSION_NUMBER,  makes  up  the  primary 
key  for  the  Models  table. 


ADDRESS:  This  field  is  the  address  of  the  office  or  unit  responsible  for  maintaining 
the  MS&A  tool. 

NAME:  OPR_ADDRESS 

LENGTH:  200 
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TYPE; 


Text 


AF  HIERARCHY :  This  field  uses  the  five  levels  of  the  Air  Force  Hierarchy  as  a  selective 
criteria.  The  five  levels  of  the  AF  Hierarchy  are  “(1)  Strategic/National  Military 
Strategy  level,  (2)  Theater/  Campaign  level,  (3)  Mission  level,  (4)  Engagement/ 
Submission  level,  and  (5)  System/  subsystem  component  (engineering)  level.” 

(17:  3-16)  There  are  five  fields  which  contribute  to  the  AF  Hierarchy  as  follows: 


NAME: 

LENGTH: 

TYPE: 

MODEL_AF_HIER_STRATEGY 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_AF_HIER_THEATER 
•  1 

Yes/No 

NAME: 

LENGTH: 

TYPE; 

MODEL_AF_HIER_MISSON 

1 

Yes/No 

NAME; 

LENGTH: 

TYPE: 

MODEL_AF_HIER_ENGAGE 

1 

Yes/No 

NAME; 

LENGTH: 

TYPE: 

MODEL_AF_HIER_SYSTEM 

1 

Yes/No 

CLASS  OF  SIMULATION:  This  field  is  the  categorization  of  simulation  into  live, 

virtual-  human,  virtual-prototype,  and  constructive  designations.  Live  simulation 
involves  real  people  operating  real  systems.  Virtual-human  means  a  virtual 
simulation  with  a  human-in-the-loop  aspect.  Other  names  include  man-in-the-loop, 
warfighter-in-  the-loop,  or  person- in-the-loop.  Virtual-prototype  is  the  interface 
of  a  realistic  computer  simulation  within  a  synthetic  environment.  Constructive 
simulation  is  simulations  that  involve  simulated  people  operating  simulated 
systems.  There  are  four  fields  which  contribute  to  the  Class  of  Simulation  as 
follows: 

NAME;  MODEL_CLASS_LIVE 

LENGTH:  1 

TYPE:  Yes/No 

NAME;  MODEL_CLASS_VIRTUAL_HUMAN 
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LENGTH: 

1 

TYPE: 

Yes/No 

NAME: 

MODEL_CLASS_VIRTUAL_PROTO 

LENGTH: 

1 

TYPE: 

Yes/No 

NAME: 

MODEL_CLASS_CONSTRUCnVE 

LENGTH: 

1 

TYPE: 

Yes/No 

COMMERCIAL  PHONE: 

This  field  is  the  commercial  phone  number  of  the  office  or  unit 

responsible  for  maintaining  the  MS&A  tool. 

NAME: 

OPR_COMMPHONE 

LENGTH: 

15 

TYPE; 

Text 

COMMON-USE  M&S:  This  field  describes  the  M&S  applications,  services,  or  materials 

that  can  apply  or  be  provided  by  a  DoD  Component  to  two  or  more  DoD 

Components. 

NAME: 

MODEL_COMMON 

LENGTH: 

Up  To  64,000 

TYPE: 

Memo 

DATA  VERIFICATION,  VALIDATION,  &  CERTIFICATION  (VV&C).  These  fields 

represent  the  Data  VV&C  status  of  the  data  that  went  into  developing  the  model. 
Data  VV&C  is  “the  process  of  verifying  the  internal  consistency  and  correctness  of 

data,  validating  that  it  represents  real  world  entities  appropriate  for  its  intended 
purpose  or  an  expected  range  of  purposes,  and  certifying  it  as  having  a  specified 
level  of  quality  or  as  being  appropriate  for  a  specified  use,  type  of  use,  or  range  of 
uses.”  (27:  x)  There  are  two  fields  which  contribute  to  the  Data  VV&C  status 

as  follows: 

NAME: 

MODEL_DATA_VVC_CODE 

LENGTH: 

2 

TYPE: 

Text 

NOTES: 

Must  enter  either  “Ve”  for  verification,  “Va”  for  validation, 
or  “Ce”  for  certification. 

NAME: 

MODEL_DATA_VVC_DATE 
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LENGTH:  8 

TYPE:  Date/Time 


DESCRIPTION:  This  field  is  a  text  entry  that  describes  the  program  and  highlights  any 
additional  information  that  the  OPR  feels  is  important  and  not  contained  in  other 
fields. 


NAME:  MODEL_DESCRIP 

LENGTH:  Up  to  64,000 

TYPE:  Memo 


DISTRIBUTED  OPERATION:  This  field  is  a  text  entry  that  describes  how  the 
“operation  of  the  simulation  can  be  spread  across  several  platforms,  if  need  be, 
particularly  to  collocate  simulation  assets  with  geographically  dispersed  users.”  (27:  A-2) 

NAME:  MODEL_DISTRIBUTED_OPS 

LENGTH:  Up  to  64,000 

TYPE:  Memo 


DUTY  ORGANIZATION:  This  field  is  the  name  of  the  duty  organization  responsible  for 
maintaining  the  MS&A  tool.  This  field  is  found  in  both  the  Models  and  Office 
tables. 


NAME:  MODEL_DUTYORG 

LENGTH:  75 

TYPE:  Text 

NOTES:  This  field,  in  conjunction  with 

MODEL_OFFICE_SYMBOL,  makes  up  the  foriegn  key 
for  the  Models  table. 

NAME:  OPR_DUTYORG 

LENGTH:  75 

TYPE:  Text 

NOTES:  This  field,  in  conjunction  with  OPR_OFFICE_SYMBOL, 

makes  up  the  primary  key  for  the  Office  table. 


DUTY  PHONE:  This  field  is  the  phone  number  of  the  office  or  unit  responsible  for 
maintaining  the  MS&A  tool. 

NAME:  OPR.DUTYPHONE 

LENGTH:  15 
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TYPE: 


Text 


EMAE.  ADDRESS:  This  field  is  the  E-mail  address  of  the  office  or  unit  responsible  for 
maintaining  the  MS&A  tool. 

NAME:  OPR_EMAIL 

LENGTH:  50 

TYPE:  Text 


EXTENSION:  This  field  provides  an  extension  or  multiple  extension  capability  for  the 
phone  number  of  the  office  or  unit  responsible  for  maintaining  the  MS&A  tool 

NAME:  OPR_EXTENSION 

LENGTH:  25 

TYPE:  Text 

FAX,  COMMERCIAL:  This  field  is  the  commercial  fax  number  of  the  office  or  unit 
responsible  for  maintaining  the  MS&A  tool. 

NAME:  OPR_FAXCOMM 

LENGTH:  15 

TYPE:  Text 


FAX,  DSN:  This  field  is  the  DSN  fax  number  of  the  office  or  unit  responsible  for 
maintaining  the  MS&A  tool. 

NAME:  OPR_FAXDSN 

LENGTH:  15 

TYPE:  Text 


FIDELITY:  This  field  describes  “the  accuracy  of  the  representation  when  compared  to 
the  real-world.”  (27:  xii) 

NAME:  MODEL_FIDELITY 

LENGTH:  Up  to  64,000 

TYPE:  Memo 


FUNCTIONAL  AREA  OF  APPLICATION:  This  field  represents  the  division  of  the  user 
community  into  the  following  functional  areas;  education,  training,  military 
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operations,  research  and  development,  test  and  evaluation,  analysis,  production, 
and  logistics.  Education,  training,  and  military  operations  is  concerned  with  the  re¬ 
creation  of  historical  battles,  doctrine  and  tactics  development,  command  and  unit 
training,  operational  planning  and  rehearsal,  and  wartime  situation  assessment. 
Research  and  development  looks  at  requirements  definition,  engineering  design 
support  and  systems  performance  assessment.  Test  and  evaluation  is  the  early 
operational  assessment,  development  and  operational  test  design;  and  operational 
excursions  and  post-test  analysis.  Analysis  focuses  on  campaign  analysis,  force 
structure  assessment,  system  configuration  determination,  sensitivity  analysis  and 
cost  analysis.  Production,  logistics,  and  design  is  concerned  with  system 
producibility  assessment,  industrial  base  appraisal,  and  logistics  requirements 
determination.  There  are  nine  fields  which  contribute  to  Functional  Area  of 
Application  as  follows: 


NAME: 

LENGTH: 

TYPE: 

MODEL_FUNC_AREA_EDUC 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

M0DEL_FUNC_AREA_TRA1N 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_FUNC_AREA_MIL_OPS 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_FUNC_AREA_ANAL 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_FUNC_AREA_R&D 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_FUNC_AREA_T&E 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_FUNC_AREA_PROD 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_FUNC_AREA_LOG 

1 

Yes/No 
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NAME:  MODEL_FUNC_AREA_DESIGN 

LENGTH:  1 

TYPE:  Yes/No 


INTERNAL  CHARACTERISTICS:  This  field  is  based  upon  certain  characteristics  that 
could  describe  the  internal  operation  or  method  of  computation  within  a  simulation 
model.  There  are  six  characteristics;  Static,  Dynamic,  Deterministic,  Stochastic, 
Discrete,  and  Continuous.  A  static  simulation  model  is  a  representation  of  a 
system  at  a  particular  time,  whereas  a  dynamic  simulation  model  is  a  representation 
of  a  system  as  it  evolves  over  time.  (11:3)  A  simulation  model  is  said  to  be 
deterministic  if  it  contains  no  random  variables  and  it  is  considered  to  be  stochastic 
if  it  contains  one  or  more  random  variables.  (11:3)  Discrete-event  simulation 
concerns  the  modeling  of  a  system  as  it  evolves  over  time  by  a  representation  in 
which  the  state  variables  change  only  at  a  countable  number  of  points  in  time. 
(11:4)  Continuous  simulation  concerns  the  modeling  over  time  of  a  system  by  a 
representation  in  which  the  state  variables  change  continuously  with  respect  to 
time.  (11:46)  There  are  six  fields  which  contribute  to  Internal  Characteristics  as 
follows: 


NAME: 

LENGTH: 

TYPE: 

MODEL_STATIC 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_DYNAMIC 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_DETERMIN 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_STOCHASTIC 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_DISCRETE 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_CONHNUOUS 

1 

Yes/No 
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LANGUAGE:  This  field  describes  the  language  the  model  is  written  in  and  any  special 
input  criteria. 

NAME:  MODEL_LANGUAGE 

LENGTH:  Up  to  64,000 

TYPE:  Memo 


LEGACY  INTERFACE:  This  is  a  yes/no  field  that  reflects  whether  or  not  a  “new 
simulations  will  interoperate  with  some  selected  set  of  existing  simulations.” 
(27:  A-2) 

NAME:  MODEL_LEGACY 

LENGTH:  1 

TYPE:  Yes/No 


LIMITATIONS:  This  field  is  a  text  entry  on  describing  the  limiting  factors  that  affects  the 
operation  of  the  program. 

NAME:  MODEL_LIMITS 

LENGTH:  Up  to  64,000 

TYPE:  Memo 


M&S  INTEROPERABILITY:  This  field  is  a  text  entry  that  defines  “the  ability  of  a  model 
or  simulation  to  provide  services  to,  and  accept  services  from,  other  models  and 
simulations,  and  to  use  the  services  so  exchanged  to  enable  them  to  operate 
effectively  together.”  (27:  xv) 

NAME:  MODEL_M&S_INTEROP 

LENGTH:  Up  to  64,000 

TYPE:  Memo 


MILITARY  CAPABILITY.  This  field  is  a  selection  criteria  based  on  the  Four  Pillars  of 
Military  Capability,  it  is  suggested  that  M&S  can  substantially  improve  capability 
and  decision  making  in  each  of  the  four  pillars  of  military  capability:  (1)  readiness, 
(2)  modernization,  (3)  force  structure,  and  (4)  substainability.  There  are  four 
fields  which  contribute  to  Military  Capability  as  follows: 

NAME:  MODEL_MILCAP_READINESS 
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LENGTH: 

1 

TYPE: 

Yes/No 

NAME: 

MODEL_MILCAP_MODERN 

LENGTH: 

1 

TYPE: 

Yes/No 

NAME: 

MODEL_MILCAP_FORCE 

LENGTH: 

1 

TYPE: 

Yes/No 

NAME: 

MODEL_MILCAP_SUSTAIN 

LENGTH: 

1 

TYPE: 

Yes/No 

MODEL  VERIFICATION,  VALIDATION,  AND  ACCREDITATION  (VV&A):  This 
field  displays  the  Verification,  Validation,  and/or  Accreditation  of  a  model 
Accreditation  is  “the  official  certification  that  a  model  or  simulation  is  acceptable 
for  use  for  a  specific  purpose.”  (27 :  viii)  Validation  is  “the  process  of 
determining  the  extent  to  which  a  model  or  simulation  is  an  accurate  representation 
of  the  real  world  from  the  perspective  of  the  intended  use(s)  of  the  model  or 
simulation.”  (27:  xvi)  Verification  is  “the  process  of  determining  that  a  model 
or  simulation  implementation  accurately  represents  the  developer’s  conceptual 
description  and  specification.  Verification  also  evaluates  the  extent  to  which  the 
model  or  simulation  has  been  developed  using  sound  and  established  software 
engineering  techniques.”  (27:  xvii)  There  are  six  fields  which  describe  the  Model 
Verification,  Validation,  and  Accreditation  as  follows: 


NAME: 

MODEL_VER_DATE 

LENGTH: 

8 

TYPE: 

Date/Time 

NAME: 

MODEL_VER_AGENCY 

LENGTH: 

55 

TYPE: 

Text 

NAME: 

MODEL_VAL_DATE 

LENGTH: 

8 

TYPE: 

Date/Time 

NAME: 

MODEL_VAL_AGENCY 

LENGTH: 

55 

TYPE: 

Text 
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NAME;  MODEL_ACCRED_DATE 

LENGTH:  8 

TYPE:  Date/Time 

NAME:  MODEL_ACCRED_AGENCY 

LENGTH:  55 

TYPE:  Text 


OFFICE  SYMBOL;  This  field  is  the  office  symbol  of  the  unit  responsible  for  maintaining 
the  MS&A  tool.  This  field  is  found  in  both  the  Models  and  Office  tables. 

NAME:  MODEL_OFFICE_SYMBOL 

LENGTH:  10 

TYPE:  Text 

NOTES:  This  field,  in  conjunction  with 

MODEL_DUTYORG,  makes  up  the  foriegn  key  for 
the  Models  table. 

NAME:  OPR_OFHCE_SYMBOL 

LENGTH;  10 

TYPE:  Text 

NOTES:  This  field,  in  conjunction  with 

OPR_DUTYORG,  makes  up  the  primary  key  for 
the  Office  table. 


PORTABILITY:  This  field  is  a  text  entry  which  discusses  how  a  “simulation  can  be  run 
on  a  variety  of  computing  platforms.”  (27;  A-2) 

NAME:  MODEL_PORTABILITY 

LENGTH:  Up  to  64,000 

TYPE:  Memo 

REMARKS:  This  field  provides  the  OPR  the  capability  of  adding  remarks  such  as  names 
or  other  pieces  of  important  information. 

NAME:  OPR_REMARKS 

LENGTH:  250 

TYPE:  Text 


REUSE:  This  field  is  a  text  entry  that  describes  which  “components  of  one  simulation  can 
be  used  in  another  appropriate  simulation.”  (27:  A-2) 


NAME:  MODEL_REUSE 

LENGTH:  Up  to  64,000 

TYPE:  Memo 


ROLES  OF  AEROSPACE  POWER:  This  field  is  a  potential  discriminator  of  the  various 
types  of  MS&A  which  could  be  useful  for  organization  metrics.  There  are  four 
roles  of  aerospace  power;  Aerospace  Control,  Force  Application,  Force 
Enhancement,  and  Force  Support.  There  are  four  fields  which  describe  Roles  of 
Aerospace  Power: 


NAME: 

MODEL_ROLES_CONTROL 

LENGTH: 

1 

■  TYPE: 

Yes/No 

NAME: 

MODEL_ROLES_APPL 

LENGTH: 

1 

TYPE: 

Yes/No 

NAME: 

MODEL_ROLES_ENHANCE 

LENGTH: 

1 

TYPE: 

Yes/No 

NAME: 

MODEL_ROLES_SUPPORT 

LENGTH: 

1 

TYPE: 

Yes/No 

RUN  TIME:  This  field  describes  how  long  it  takes  to  mn  the  simulation;  i.e.  minutes^ 

hours,  days,  or  weeks. 

NAME: 

MODEL_RUN_TIME 

LENGTH: 

Up  to  64,000 

TYPE: 

Memo 

SECURITY  LEVEL:  This  field  identifies  the  Security  level  of  the  model. 

NAME:  MODEL_SECURITY_LEVEL 

LENGTH:  2 

TYPE:  Text 

NOTES :  Must  enter  either  “Un”  for  unclassified,  “CN”  for 

confidential,  “SE”  for  secret,  “TS”  for  top  secret,  “FO”  is 
for  official  use  only,  “NO”  for  no  foreign  nationals. 
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“LD”  for  limited  distribution,  and  “SI”  for  secret 
compartmented  information. 


STANDALONE:  This  field  shows  whether  or  not  the  MS&A  tool  can  stand  alone  or  can 
only  provide  data  when  incorporated  with  another  model. 

NAME:  MODEL_STANDALONE 

LENGTH:  1 

TYPE:  Yes/No 


SYSTEM  SEGMENT.  This  field  is  an  added  discriminator  for  the  “Weapon  System” 
category.  For  instance,  a  user  might  be  interested  in  fighter  communication 
systems  or  fighter  missiles.  This  field  would  include  descriptors  such  as  avionics, 
navigation,  radar,  communications,  weapons,  missiles,  intelligence,  computers, 
cost,  propulsion,  structure,  and  other.  Conversely,  this  field  could  be  used  as  a 
selection  criteria  in  its  own  as  well.  There  are  twelve  fields  contained  in  System 
Segment  as  follows: 

NAME:  MODEL_SYSTEM_AVIONIC 

LENGTH:  1 

TYPE:  Yes/No 

NAME:  MODEL_SYSTEM_NAV 

LENGTH:  1 

TYPE:  Yes/No 

NAME:  MODEL_SYSTEM_RADAR 

LENGTH:  1 

TYPE:  Yes/No 

NAME:  MODEL_SYSTEM_COMM 

LENGTH:  1 

TYPE:  Yes/No 

NAME:  MODEL_SYSTEM_WEAPON 

LENGTH:  1 

TYPE:  Yes/No 

NAME:  MODEL_SYSTEM_MISSILE 

LENGTH:  1 

TYPE:  Yes/No 

NAME:  MODEL_SYSTEM_INTEL 
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LENGTH:  1 

TYPE:  Yes/No 

NAME:  MODEL_SYSTEM_COMPUTER 

LENGTH:  1 

TYPE:  Yes/No 

NAME:  MODEL_SYSTEM_COST 

LENGTH:  1 

TYPE:  Yes/No 

NAME:  MODEL_SYSTEM_PROPULSION 

LENGTH:  1 

TYPE:  Yes/No 

NAME:  MODEL_SYSTEM_STRUCTURE 

LENGTH:  1 

TYPE:  Yes/No 

NAME:  MODEL_SYSTEM_OTHER 

LENGTH:  1 

TYPE:  Yes/No 


TITLE:  This  field  is  the  tide  of  the  MS&A  tool. 

NAME:  MODEL_TITLE 

LENGTH:  100 

TYPE:  Text 

NOTES:  This  field,  in  conjunction  with  MODEL_ACRONYM  and 

MODEL_VERSION_NUMBER,  makes  up  the  primary  key 
for  the  Models  table. 

TYPE  OF  M&S:  This  field  is  a  selective  one  based  on  the  “three  general  types  of  models 
which  are;  wargaming,  training,  and  acquisition.  Wargaming  models  range  from 
single  engagement  (one-on-one)  to  joint  theater  level  campaign  operations. 
Training  models  range  from  single  template  instructional  systems  to  complex 
virtual  reality  simulations.  Acquisition  models  range  from  physical  level 
phenomenon  models  through  engineering  component  design  tools  to  models  of 
systems- in-the-end-use-environment.”  (17: 1-3)  There  are  three  fields  which 
contribute  to  Types  of  M&S  as  follows: 

NAME:  MODEL_TYPE_WARGAME 

LENGTH:  1 
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TYPE: 

Yes/No 

NAME: 

MODEL_TYPE_TRAIN 

LENGTH: 

1 

TYPE: 

Yes/No 

NAME: 

MODEL_TYPE_ACQ 

LENGTH: 

1 

TYPE: 

Yes/No 

TYPES  OF  PLATFORMS: 

This  field  describes  the  type  of  platform  or  hardware 

requirements  needed  to  run  the  model.  Main  frames,  personal  computer,  or 
workstations  as  some  examples. 

NAME: 

MODEL.PLATFORM 

LENGTH: 

Up  to  64,000 

TYPE: 

Memo 

VERSION  NUMBER:  This  field  is  the  version  number  of  the  MS&A  tool. 

NAME: 

MODEL_VERSION_NUMBER 

LENGTH: 

8 

TYPE: 

Number  (Double) 

NOTES: 

This  field,  in  conjunction  with  MODEL_ACRONYM  and 

MODEL_TITLE,  makes  up  the  primary  key  for  the 

Models  table. 

WEAPON  SYSTEM:  This  field  is  a  high  level  descriptor  of  the  entity  that  the  model  is 

capable  of  depicting.  Entities  identified  are  fighters,  tankers,  transports,  bombers, 
helicopters,  satellites,  missiles,  command  and  control,  and  others.  There  are  nine 

fields  which  describe  Weapon  System  as  follows: 

NAME: 

MODEL_WEAPON_FIGHTER 

LENGTH: 

1 

TYPE: 

Yes/No 

NAME: 

MODEL_WEAPON_TANKER 

LENGTH: 

1 

TYPE: 

Yes/No 

NAME: 

MODEL_WEAPON_TRANSPORT 

LENGTH: 

1 
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TYPE: 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_WEAPON_BOMBER 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_WEAPON_HELI 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_WEAPON_SATELnE 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_WEAPON_MISSILE 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_VVEAPON_COMAND 

1 

Yes/No 

NAME: 

LENGTH: 

TYPE: 

MODEL_WEAPON_OTHER 

1 

Yes/No 
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APPENDIX  F 


QUESTIONNAIRE 

This  questionnaire  is  designed  to  solicit  your  inputs  on  the  MS&A  Prototype  Database. 
Your  responses  are  purely  voluntary  and  greatly  appreciated. 

What  is  your  AFSC  or  Civilian  Code? _ 

What  is  your  rank  or  grade? _ 

How  long  have  you  been  associated  with  M&S?  _ 


Please  think  back  to  your  last  M&S  experience  and  the  circumstances  or 
requirements  associated  with  that  experience. 

What  was  the  title  of  the  M&S  model  or  tool  that  you  used?  _ 


Now  based  on  that  experience,  try  using  the  database  to  ascertain  what  models  or 
tools  that  are  available  to  address  your  area  of  application. 


Question  1:  Did  your  model  or  tool  appear  in  the  suggested  list?  YES  NO 

Question  2:  For  each  of  the  suggested  models  or  tools,  please  apply  a  usefulness  rating 
as  to  whether  or  not  that  particular  model  or  tool  would  be  useful  in  your  application. 
The  usefulness  ratings  are  as  follows: 

1  =  Extremely  Useful 
3  =  Marginally  Useful 
5  =  Not  Useful  At  All 


MODEL  TITLE 


USEFULNESS 
1  2  3  4  5 

1  2  3  4  5 


105 


MODEL  TITLE  USEFULNESS 

_ 1  2  3  4  5 

1  2  3  4  5 

_ 1  2  3  4  5 

1  2  3  4  5 

_ 1  2  3  4  5 

■  1  2  3  4  5 

1  2  3  4  5 

_ 1  2  3  4  5 

_  1  2  3  4  5 

_ 1  2  3  4  5 

Question  3;  How  would  rate  the  usefulness  of  this  database  (using  the  same  scale)? 

1  2  3  4  5 

Thank  you  for  your  assistance.  If  you  should  have  any  comments  or  suggestions, 
please  feel  free  to  use  the  space  below. 
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